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First Things First 


Welcome to Linux! If youre a new user, this book can serve as a 
quick introduction, as well as a guide to common and practical 
commands. If you have Linux experience, feel free to skip the 
introductory material. 


What’s in This Book? 


This book is a short guide, not a comprehensive reference. I 
cover important, useful aspects of Linux so you can work pro- 
ductively. I do not, however, present every single command 
and every last option (my apologies if your favorite was omit- 
ted), nor delve into detail about operating system internals. 
Short, sweet, and essential—that’s our motto. 


I focus on commands, those pesky little words you type on a 
command line to tell a Linux system what to do. Here’s an 
example command that counts lines of text in a file, myfile: 


wc -l myfile 


This book covers important Linux commands for most users, 
such as 1s (list files), grep (search for text), mplayer (play audio 
and video files), and df (measure free disk space). I touch 
only briefly on graphical environments like GNOME and KDE 
Plasma, each of which could fill a Pocket Guide by itself. 


I've organized the material by function to provide a concise 
learning path. For example, to help you view the contents of a 
file, I introduce many file-viewing commands together: cat for 
short text files, less for longer ones, od for binary files, and so 
on. Then I explain each command in turn, briefly presenting its 
common uses and options. 


I assume you have access to a Linux system and can log in. 
If not, it’s easy to try out Linux on most computers. Just 
download and install a “live” Linux distribution onto a USB 
thumb drive and boot it. Examples are Ubuntu, Fedora, and 
KNOPPIX. 


What’s New in the Fourth Edition? 


New commands 
Tve added 50 new commands to this edition, such as git 
and svn for version control, split and column for text 
manipulation, pandoc and ffmpeg for file conversion, snap 
and flatpak for package management, mdadm, lvcreate, 
and zfs for fancy storage management, gpg for encryp- 
tion, and many others. 


Clearer organization 
I’ve reorganized the book into chapters on concepts, files, 
basic system administration, networking, and other topics. 


Goodbye, ancient commands 
Some commands from previous editions of this book are 
mostly obsolete today, such as write and finger, or depre- 
cated, such as ftp. I’ve replaced them with more relevant 
commands for modern Linux systems. 


Conventions Used in This Book 


Each command I present in this book begins with a standard 
heading. Figure P-1 shows the heading for 1s, a command 
that lists the names and attributes of files. The heading demon- 
strates the command's general usage in a simple format: 
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ls [options] [files] 


which means youd type “Is” followed, if you choose, by options 
and then filenames. Don’t type the square brackets “[” and “]” 
—they just indicate their contents are optional. Words in italics 
mean you have to fill in your own values, like names of actual 
files. If you see a vertical bar between options or arguments, 
perhaps grouped by parentheses: 


(file | directory) 


this indicates choice: you may supply either a filename or 
directory name as an argument. 


Is stdin stdout -file --help --version 


\s [options] [files] 


Figure P-1. Standard command heading 


The standard heading shown in Figure P-1 also includes six 
properties of the command, printed in black (supported) or 
gray (unsupported): 


stdin 
The command reads by default from standard input (i-e., 
your keyboard). See “Input, Output, and Redirection” on 
page 43. 


stdout 
The command writes by default to standard output (ce., 
your display). See “Input, Output, and Redirection” on 
page 43. 


- file 
A single-dash argument (-), when provided as an input 
filename, tells the command to read from standard input 
rather than a disk file. Likewise, if the dash is supplied 
as an output filename, the command writes to standard 
output. For example, the following wc command line reads 
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the files myfile and myfile2, then standard input, then 
myfile3: 


wc myfile myfile2 - myfile3 
-- opt 


A double-dash option (--) means “end of options”: any 
strings appearing later on the command line are not 
treated as options. A double dash is sometimes necessary 
to work with a filename that begins with a dash, which 
otherwise would be (mistakenly) treated as an option. For 
example, if you have a file named -dashfile, the command 
wc -dashfile fails because the string -dashfile is treated 
as an (invalid) option. Run wc -- -dashfile to indicate 
-dashfile is a filename. If a command does not support 
“..”, you can still work around the problem by prepending 
the current directory path “./” to the filename so the dash 
is no longer the first character: 


we ./-dashfile 


--help 
The option --help makes the command print a help mes- 
sage explaining proper usage, then exit. 


--version 
The option --version makes the command print its ver- 
sion information and exit. 


Commands, Prompts, and Output 


The Linux command line, or shell, prints a special symbol, 
called a prompt, when it’s waiting for a command. In this book, 
the prompt is a right-facing arrow: 


> 


Prompts come in all shapes and sizes, depending on how 
your shell is configured. Your prompt might be a dollar sign 
($), a combination of your computer name, username, and 
various symbols (myhost:~smith$), or something else. Every 
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prompt means the same thing: the shell is ready for your next 
command. 


When I show a command line in this book, some parts are 
meant to be typed by the user, and other parts are not (like the 
prompt and the command’s output). I use boldface to identify 
the parts to type. Sometimes I add italic comments to explain 
what’s going on: 


> we -l myfile The command to type at the prompt 
18 myfile ‘The output it produces 


Your Friend, the echo Command 


In many of my examples, I print information to the screen 
with the echo command, which I formally describe in “Screen 
Output” on page 275. echo is one of the simplest commands—it 
merely prints its arguments on standard output, once those 
arguments have been processed by the shell: 


> echo My dog has fleas 
My dog has fleas 
> echo My name is SUSER The shell variable USER 


My name is smith 


Long Command Lines 


Sometimes, a command is longer than the width of a page, so 
I split it onto multiple lines. A final backslash character means 
“continued on the next line”: 


> echo This is a long command that does not fit on \ 
one line 
This is a long command that does not fit on one line 


If you enter one of my multiline commands in a running shell, 
feel free to break it up with backslashes as I did, or just type the 
whole command on one line without backslashes. 
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Keystrokes 


I use certain symbols for keystrokes. The caret (*) means “hold 
down the Control key; usually labeled Ctrl. For example, *D 
(Ctrl D) means “hold down the Ctrl key and type D? I also 
write ESC to mean “press and release the Escape key.’ Keys like 
Enter and the space bar should be self-explanatory. 


Downloading the Practice Files 


I’ve created a collection of files to help you practice with Linux. 
Download and install them on any Linux machine, and you 
can run most of the example commands in this book verbatim. 
To download them for the first time, run the following com- 
mands. (Note that -0 contains a capital O, not a zero.) 


> cd 
> curl -0 https://linuxpocketguide.com/LPG4.tar.gz 
> tar -xf LPG4.tar.gz 


The preceding commands create a directory named linuxpock- 
etguide in your home directory. Visit this directory: 


> cd ~/linuxpocketguide 


and run commands as you read the book. The output should 
match the book’s except for local details like dates and 
usernames. 


To re-download and install the practice files (say, if you've 
modified them), simply run the provided reset- lpg script: 


> cd ~/linuxpocketguide 
> bash reset-lpg 


1 Or, if you are experienced with git and GitHub, download the files 
and skip the rest of my instructions. If you clone the repository and 
want to restore the files to their original state, don't run the reset-1pg 
script; run git reset --hard instead. 
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If you've placed the practice files in a different directory, supply 
it to reset-lpg. The following command creates or refreshes 
the directory /tmp/practice/linuxpocketguide: 


> bash reset-lpg /tmp/practice 


Conventions Used in This Book 


The following typographical conventions are used in this book: 


Italic 
Indicates new terms, URLs, email addresses, filenames, 
and file extensions. 


Constant width 
Used for program listings, as well as within paragraphs 
to refer to program elements such as variable or function 
names, databases, data types, environment variables, state- 
ments, and keywords. 


Constant width bold 
Shows commands or other text that should be typed liter- 
ally by the user. 


Constant width italic 
Shows text that should be replaced with user-supplied 
values or by values determined by context. 


TIP 


This element signifies a tip or suggestion. 


NOTE 


This element signifies a general note. 


First Things First | xv 


WARNING 


This element indicates a warning or caution. 


O'Reilly Online Learning 
For more than 40 years, O'Reilly Media 


e ® 
O R = | L LY has provided technology and business 
training, knowledge, and insight to help 
companies succeed. 


Our unique network of experts and innovators share their 
knowledge and expertise through books, articles, and our 
online learning platform. O’Reilly’s online learning platform 
gives you on-demand access to live training courses, in-depth 
learning paths, interactive coding environments, and a vast 
collection of text and video from O'Reilly and 200+ other pub- 
lishers. For more information, visit https://oreilly.com. 


How to Contact Us 


Please address comments and questions concerning this book 
to the publisher: 


O'Reilly Media, Inc. 

1005 Gravenstein Highway North 

Sebastopol, CA 95472 

800-889-8969 (in the United States or Canada) 
707-827-7019 (international or local) 
707-829-0104 (fax) 

support@oreilly.com 
https://www.oreilly.com/about/contact.html 


We have a web page for this book, where we list errata, exam- 
ples, and any additional information. You can access this page 
at https://oreil.ly/linux-pocket-guide-4e. 
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For news and information about our books and courses, visit 
https://oreilly.com. 


Find us on LinkedIn: https://linkedin.com/company/oreilly- 
media. 
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CHAPTER 1 
Essential Concepts 


What’s Linux? 


Linux is a free, open source operating system (OS) that’s an 
alternative to Microsoft Windows and Apple macOS. Linux 
powers most of the servers on the internet. It operates behind 
the scenes on every Android mobile phone and Chromebook, 
and on millions of network-connected devices like routers, 
firewalls, and robotic cow-milking systems (seriously). It also 
runs fine on desktop and laptop computers. 


Linux has four major parts, shown in Figure 1-1: 


The kernel 
Low-level software. It controls the hardware and basic 
functions like process scheduling and networking. Few 
users interact with the kernel directly. 


Supplied programs 
Thousands of programs for file handling, text edit- 
ing, software development, web browsing, audio, video, 
encryption, mathematics...you name it. These programs 
talk to the kernel. Programs that run on the command line 
are called commands. 
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The shell 

A Linux program for running commands and displaying 
the results. Linux has an assortment of shells with differ- 
ent features. This book focuses on a shell called bash, 
which is often the default for user accounts. Some other 
shells are dash, fish, ksh (Korn shell), tcsh (TC shell, or 
T shell), zsh (Z shell), and to a lesser extent, busybox. All 
shells have similar functions, though their usage varies. 


Graphical desktop environment (optional) 
A UI with windows, menus, icons, mouse support, and 
other familiar GUI elements. Some popular environments 
are GNOME and KDE Plasma. Most applications built for 
GNOME can run in KDE and vice versa.! 


This book focuses on the command-line parts of Linux, namely 
the supplied programs and the shell. Windows and macOS 
have command-line interfaces too (cmd and powershell on 
Windows, Terminal on the Mac), but most of their users stick 
with the GUI and might never see or need a command line. On 
Linux, the shell is critical. If you use Linux without the shell, 
you are missing out. 


Linux is extremely configurable and comes in hundreds of vari- 
eties that serve different needs and tastes. Each variety is called 
a distro (short for “distribution”). All distros share some core 
components but may look different and include different pro- 
grams and files. Some popular distros include Mint, Ubuntu, 
Manjaro, Arch, Gentoo, Red Hat, and OpenSUSE, among oth- 
ers. The core material in this book should apply to every distro. 


1 GNOME, KDE, and other environments are built on a common win- 
dowing system that is either X or Wayland. To see which system youre 
using, run the command echo $XDG_SESSION_TYPE. 
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Kernel 


71 Call 


Programs 
(commands) 


Graphical desktop 
environment 


Figure 1-1. The four major parts of Linux, conceptually. Low-level 
kernel functions are called by programs, which are invoked in a shell, 
which can be launched by a graphical desktop. 


Launching a Shell 


Where do shells come from? Sometimes Linux will launch one 
for you automatically. This is often the case when you log in 
over a network using ssh or a similar tool. The first significant 
thing you see is a shell prompt awaiting your command. 


Other times, you have to launch a shell manually. This is com- 
mon when using a graphical desktop full of icons and menus 
with no shell in sight. In such cases, you need a GUI applica- 
tion called a terminal or terminal program that runs shells in 
a window. The sidebar “Shell Versus Terminal” on page 22 
clarifies the difference between shells and terminals. 
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Every distro with a graphical desktop includes at least one 
terminal program, but you might have to hunt for it. Search for 
an application, icon, or menu item named Terminal, Konsole, 
xterm, gnome-terminal, uxterm, or something similar, and run 
it to open a terminal. Also try pressing Ctr1-Alt-t (hold the 
Control and Alt keys and press T), which opens a terminal in 
some environments. 


Shell Versus Terminal 


A shell is a command-line interface for launching Linux com- 
mands by typing plain text. It prints a prompt and waits for 
your command: 


> 


A terminal is a program that opens a window and presents a 
running shell, shown in Figure 1-2. It’s like a graphical wrapper 
around a shell. A terminal adds menus, scrollbars, copy and 
paste, and other GUI features that support the shell. 


Terminal va xXx 


File Edit View Search Terminal Help 


Figure 1-2. A terminal program opens a window that runs a shell. 


Command-Line Warm-Up 


To give you a feel for Linux, here are 10 simple commands to 
try right now in a shell. Type them exactly, including capital 
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and small letters, spaces, and all symbols after the prompt. At 
the end of each command, press Enter.” 


Display a calendar for November 2023: 


> cal nov 2023 
November 2023 

Su Mo Tu We Th Fr Sa 
1 2 3-4 

5 6 7 8 9 10 11 

12 13 14 15 16 17 18 

19 20 21 22 23 24 25 

26 27 28 29 30 


List the contents of the /bin directory, which contains many 
commands: 


> 1s /bin 
bash less rm 
bunzip2 _lessecho rmdir 


busybox lessfile rnano 


Count the number of visible items in your home directory 
(represented here by a variable, HOME, that I discuss later): 


> Ls SHOME | we -l 
8 Your value may be different 


See how much space is used on a partition of your hard disk: 


> df -h / 
Filesystem Size Used Avail Use% Mounted on 
/dev/sdb1 78G 30G 48G 61% / 


99 


Watch the processes running on your computer (press “q” to 
quit): 
> top -d1 


2 If you see an error message “command not found,’ don’t worry: a 
command probably isn't installed on your system. See “Installing Soft- 
ware Packages” on page 195. 
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Print the file /etc/hosts, which contains names and addresses of 
computers, on your default printer if you have one set up: 


> lpr /etc/hosts 
See how long you've been logged in: 


> last -1 SUSER 
smith pts/7 :0 Tue Nov 10 20:12 still logged in 


Download a file sample.pdf from this book’s website to your 
current directory, without needing a web browser: 


> curl -0 https://linuxpocketguide.com/sample. pdf 


See who owns the domain name oreilly.com (press the space bar 


«9» 


to move forward page by page, and press “q” to quit): 


> whois oreilly.com | less 
Domain Name: OREILLY.COM 
Registrar: GODADDY.COM, LLC 


Finally, clear your terminal or screen: 
> clear 


Congratulations, you are now a Linux user! 


The Structure of Commands 


A Linux command typically consists of a program name fol- 
lowed by options and arguments: 


we -l myfile 


The program name (wc, short for “word count”) refers to a 
program somewhere on disk that the shell locates and runs. 
Options, which usually begin with a dash, affect the behavior 
of the program. In the preceding command, the -1 option tells 
wc to count lines and not words. The argument myfile specifies 
the file that wc should read and process. 


Commands can have multiple options and arguments. Options 
may be given individually, or combined after a single dash: 
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we -l -w myfile Two individual options 
wc -lw myfile Combined options, same as -I -w 


though some programs are quirky and do not recognize com- 
bined options. Multiple arguments are also OK: 


we -l myfile myfile2 Count lines in two files 


Options are not standardized. They may be a single dash and 
one character (say, -1), two dashes and a word (--lines), or 
several other formats. The same option may have different 
meanings to different programs: in the command wc -l, the 
option -1 means “lines of text” but in ls -l it means “long out- 
put” Two programs also might use different options to mean 
the same thing, such as -q for “run quietly” versus -s for “run 
silently” Some options are followed by a value, such as -s 10, 
and space between them might not be required (-s10). 


Arguments are usually filenames for input or output, but they 
can be other things too, like directory names, usernames, host- 
names, IP addresses, regular expressions, or arbitrary strings. 


A command that’s just a single program with options and argu- 
ments is called a simple command. Here's a simple command 
that lists users who are logged into a Linux server:* 


> who 

silver 20 Sep 23 20:44 
byrnes pts/0 Sep 15 13:51 
barrett  pts/1 Sep 22 21:15 
silver pts/2 Sep 22 21:18 


A command can also invoke several programs at once and 
even connect programs so they interact. Here’s a command that 
connects the output of who to the input of wc, which counts 
lines of text. The result is the number of lines in the output of 
who: 


3 User “silver,” who is listed twice, is running two interactive shells at 
once. 
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> who | we -l 
4 


The vertical bar, called a pipe, makes the connection between 
who and wc. Linux experts use these sorts of combined com- 
mands, called pipelines, all the time. 


Commands can also include programming language constructs 
like variables, conditionals, and loops, which I cover in “Pro- 
gramming with Shell Scripts” on page 311. For example, a 
command might say, “run this program, write its output to a 
file of my choosing, and if any errors occur, send me an email 
with the results.” 


Users and Superusers 


Linux is a multiuser OS: multiple people can run programs 
on a single Linux computer at the same time. On a given 
computer, each user is identified by a username, like smith or 
funkydance. Each user has a separate workspace of sorts (see 
“Home Directories” on page 29) so they don’t interfere with 
one another. 


A special user named root—the superuser or administrator— 
has the privileges to do anything at all on the system. The 
superuser can create, modify, or delete any file and run any 
program. Ordinary users are restricted: they can run most 
programs, but in general, they can’t mess with other users’ stuff. 


Some commands in this book require superuser privileges. I 
precede these commands with sudo: 


> sudo superuser command goes here 


WARNING 


sudo gives you the power to destroy your Linux system. 


26 | Chapter 1: Essential Concepts 


I discuss sudo fully in “Becoming the Superuser” on page 161, 
but for now, all you need to know is that sudo gives you super- 
user powers and sometimes prompts for your password. For 
example, to count lines in a protected file called /etc/shadow, 
with and without sudo, you could run this command: 


> we -l /etc/shadow This fails 

wc: /etc/shadow: Permission denied 

> sudo we -l /etc/shadow Run with sudo 
[sudo] password: xxxxXxxxx 

51 /etc/shadow It worked! 


The Filesystem 


To make use of any Linux system, you must become comforta- 
ble with Linux files and directories (a.k.a. folders), collectively 
called the filesystem. On a graphical desktop, files and directo- 
ries are obvious on screen. In a command-line interface like 
the Linux shell, the same files and directories are still present 
but less visible, so at times you must remember which directory 
you are “in” and how it relates to other directories. You'll use 
shell commands like cd (change directory) to move between 
directories, and commands like pwd (print working directory) 
to keep track of where you are in the filesystem. 


Let’s cover some terminology. Linux files are collected into 
directories. The directories form a hierarchy, or tree, as in Fig- 
ure 1-3. One directory may contain other directories, called 
subdirectories, which may themselves contain other files and 
subdirectories, and so on, into infinity. The topmost directory 
is called the root directory and is denoted by a slash (/).4 


4 In Linux, all files and directories descend from the root. This is unlike 
Windows, in which different devices are accessed by drive letters. 
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Figure 1-3. A Linux filesystem (partial). The root directory is at the top. 
The absolute path to the “dan” directory is /home/dan. 


Linux refers to its files and directories using a “names and 
slashes” syntax called a path. For instance, the following path: 


/one/two/three/four 


refers to the root directory /, which contains a directory called 
one, which contains a directory two, which contains a directory 
three, which contains a final file or directory, four. Any path 
that begins with a slash, which descends all the way from the 
root, is called an absolute path. 


Paths don't have to be absolute—they can be relative to some 
directory other than the root. Figure 1-3 has two different 
directories named bin, whose absolute paths are /bin and /usr/ 
bin. If you simply refer to “the bin directory,’ its not clear 
which one you mean. You need more context. Any path that 
doesn't begin with a slash, like bin, is called a relative path. 


To make sense of a relative path, you need to know “where you 
are” in the Linux filesystem. This location is called your current 
directory (sometimes called “working directory” or “current 
working directory”). 


Every shell has a current directory, and when you run com- 
mands in that shell, they operate relative to its current direc- 
tory. For example, if your shell is “in” the directory /usr, and 
you refer to a relative path bin, it means /usr/bin. In general, 
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if your current directory is /one/two/three, a relative path a/b/c 
would imply the absolute path /one/two/three/a/b/c. 


Two special relative paths are named . (a single period, or 
“dot”) and .. (two periods, or “dot dot”). “Dot” means your 
shell’s current directory, and “dot dot” means its parent direc- 
tory, one level above. So if your shell’s current directory 
is /one/two/three, then . refers to this directory and .. refers 
to /one/two. 


To travel from one directory to another, use the cd command, 
which changes your shell’s current directory: 


> cd /usr/local/bin Enter the directory /usr/local/bin 


The previous cd command used an absolute path. You can 
make relative moves with cd as well: 


>cdd Enter subdirectory d of my current directory 
> cd ../mydir Go up to my parent, then into directory mydir 


File and directory names may contain most characters you 
expect: capital and small letters,> numbers, periods, dashes, 
underscores, and most symbols (but not “/’, which is reserved 
for separating directories). For efficiency, however, avoid 
spaces, asterisks, dollar signs, parentheses, and other characters 
that have special meaning to the shell. These characters require 
special treatment in filenames (see “Quoting” on page 46), 
which can be inconvenient or tedious. 


Home Directories 


Users’ personal files are usually kept in the directory /home 
(for ordinary users) or /root (for the superuser). Your home 
directory is typically /home/<your-username> (/home/smith, / 
home/funkydance, etc.). There are several ways to visit or refer 
to your home directory: 


5 Linux filenames are case-sensitive, so upper and lowercase letters are 
not equivalent. 
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cd 
With no arguments, the cd command returns you (i.e., sets 
the shell’s current directory) to your home directory. 


HOME variable 
The environment variable HOME (see “Shell Variables” on 
page 39) contains the name of your home directory: 


> echo SHOME Print the directory name 
/home/smith 
> cd SHOME/Linuxpocketguide Visit a subdirectory 


When used in place of a directory, a lone tilde is expanded 
by the shell to the name of your home directory. 


> echo ~ Print the directory name 
/home/smith 
> cd ~/linuxpocketguide Visit a subdirectory 


When the tilde is followed by a username (as in ~fred), the 
shell expands this string to be the user’s home directory: 


> cd ~fred Visit Fred's home directory, if it exists 
> pwd The “print working directory” command 
/home/fred 


System Directories 


A Linux system has tens of thousands of system directories. 
They contain OS files, applications, documentation, and just 
about everything except personal user files (which typically live 
in /home). 


Unless you're a system administrator, you'll rarely visit most 
system directories—but with a little knowledge you can under- 
stand or guess their purposes. Their names often contain three 
parts, as shown in Figure 1-4. 
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/usr/local/share/emacs 


Scope Category Application 


Figure 1-4. Directory scope, category, and application 


Directory path part 1: scope 


The scope of a directory path describes, at a high level, the 
purpose of an entire directory tree. Some common ones are: 


/ (Pronounced “root”) System files supplied with your distro 
/usr (Pronounced “user”) More system files supplied with your distro 


/usr/local (Pronounced “user local”) System files that are not supplied with 
your distro; they may be unique to your local Linux network or your 
individual computer 


There isn't a clear distinction between / and /usr in practice, 
but / is considered “lower level” and closer to the OS. 


Directory path part 2: category 


The category of a directory path in Figure 1-4 describes the 
types of files found in a directory. For example, if the category 
is lib, you can be reasonably sure that the directory contains 
library files for programming. If the category is bin, the con- 
tents are usually binary files—executable programs. 


When you precede a category like bin with a scope, you pro- 
duce paths like /bin, /usr/bin, and /usr/local/bin. A distro’s 
most fundamental system programs like ls and cat are typi- 
cally in /bin, and other system programs are in /usr/bin.® /usr/ 
local/bin contains locally installed programs not included in 
your distro. These are not hard-and-fast rules but typical cases. 


6 Some distros no longer make these distinctions. Fedora, for example, 
makes /bin a symbolic link to /usr/bin. 
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Some common categories are as follows: 


Categories for programs 


bin Programs (usually binary files) 
sbin Programs (usually binary files) for superusers 
lib Libraries of code used by programs 


Categories for documentation 


doc Documentation 
info Documentation files for emacs’s built-in help system 
man Documentation files (manual pages) displayed by the man 


command; the files are often compressed and are sprinkled with 
typesetting commands for man to interpret 


share Program-specific files, such as examples and installation 
instructions 


Categories for configuration 


etc Configuration files for the system (and other miscellaneous stuff) 
init.d Configuration files for booting Linux 
rod Configuration files for booting Linux; also rc7.d, rc2.d, ... 


Categories for programming 


include Header files for programming 

src Source code for programs 

Categories for web files 

cgi-bin Scripts/programs that run on web pages 

html Web pages 

public_html Web pages, typically in users’ home directories 
www Web pages 

Categories for display 

fonts Fonts (surprise!) 

X11 X window system files 
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Categories for hardware 


dev Device files for interfacing with disks and other hardware 
media Mount points: directories that provide access to disks 
mnt Mount points: directories that provide access to disks 


Categories for runtime files 
var Files related to the state of the computer, updated frequently 


lock Lock files, created by programs to say, “I am running”; the 
existence of a lock file may prevent another program, or another 
instance of the same program, from performing an action 


log Logfiles that track important system events, containing error, 
warning, and informational messages 

mail Mailboxes for incoming email 

run PID files, which contain the IDs of running processes; these files are 
often consulted to track or kill particular processes 

spool Files queued or in transit, such as outgoing email, print jobs, and 
scheduled jobs 

tmp Temporary storage for programs and/or people to use 


Directory path part 3: application 


The application part of a directory path (Figure 1-4), if present, 
is usually the name of a program. For example, the direc- 
tory /etc/systemd has scope root (/), category etc (configuration 
files), and application systemd. Since systemd is a service for 
configuring Linux machines, a good guess is that /etc/systemd 
contains configuration files for that service—and it does. 


Kernel-Related Directories 


Some directories support the Linux kernel, the lowest-level part 
of the Linux OS: 


The Filesystem | 33 


/boot 
Files for booting the system. The kernel lives here, typi- 
cally in /boot/vmlinuz or a file of similar name. 


/lost+ found 
Damaged files that were rescued by a disk recovery tool. 


/proc 


Files for currently running processes; for advanced users. 


/sys 
Files for kernel internals; for advanced users. 


The files in /proc and /sys provide views into the running kernel 
and have special properties. Files in /proc always appear to be 
zero-sized, read-only, and dated now, but their contents magi- 
cally contain information about the Linux kernel: 


> ls -1G /proc/version 

-r--r--r-- 1 root 0 Oct 3 22:55 /proc/version 
> cat /proc/version 

Linux version 5.15.0-76-generic ... 


Files in /sys also have misleading sizes and magical contents: 


> ls -1G /sys/power/state 

-rw-r--r-- 1 root 4096 Jul 8 06:12 /sys/power/state 
> cat /sys/power/state 

freeze mem disk 


/proc and /sys are used mostly by system programs, but feel free 
to view them. Here are some examples: 

/proc/ioports A list of your computer's input/output hardware. 

/proc/cpuinfo \nformation about your computer's processors. 


/proc/version The OS version. The uname command prints the same 
information. 


/proc/uptime System uptime: seconds elapsed since the system was last booted. 
Run uptime for a more human-readable result. 


/pro/NNN Information about the Linux process with ID NWN, where NNN is a 
positive integer, such as /proc/13542. 
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/proc/self Information about the current process you're running; a symbolic 
link to a /proc/nnn file, automatically updated. Try running: 
> 1s -l /proc/self 


several times in a row, and /proc/self changes where it points. 


File Permissions 


A Linux system may have many user accounts. To maintain 
privacy and security, most users can access only some files on 
the system, not all. This access control is embodied in two 
questions: 


Who has permission? 
Every file and directory has an owner who can do any- 
thing they want with it. Typically, a file’s owner is the user 
who created it. A superuser can change a file’s owner. 


Additionally, a predefined group of users may access a 
file. Groups are defined by the system administrator and I 
cover them in “Group Management” on page 192. 


Finally, a file or directory can be opened to all users who 
have accounts on the system. You'll also see this set of 
users called the world or simply other. 


What kind of permission is granted? 
File owners, groups, and the world may each have permis- 
sion to read, write (modify), and execute (run) particular 
files. Permissions also extend to directories, which users 
may read (view files within the directory), write (create 
and delete files within the directory), and execute (enter 
the directory with cd). 


To see the ownership and permissions of a file named myfile, 
run ls -1, described in “Basic File Operations” on page 61: 


> 1s -l myfile 
-rw-r--r-- 1 smith smith 1168 Oct 28 2015 myfile 
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To see the ownership and permissions of a directory named 
mydir, add the -d option: 


> 1s -ld mydir 
drwxr-x--- 3 smith smith 4096 Jan 08 15:02 mydir 


In the output, the file permissions are the 10 leftmost charac- 
ters, a string of r (read), w (write), x (execute), other letters, and 
dashes. For example: 


- TWXP-X--- 


Here’s what these letters and symbols mean, briefly: 


Position Meaning 


1 File type: - = file, d = directory, L = symbolic link, p = 
named pipe, c = character device, b = block device 


2-4 Permissions for the file's owner: read (r), write (w), 
execute (x), or no permission (-). 

5-7 Permissions for the file’s group: r, w, x, - 

8-10 Permissions for all other users: r, w, x, - 


My example -rwxr-x--- means a file that can be read, written, 
and executed by the owner; read and executed by the group, 
but not written; and not accessed at all by other users. To 
change the owner, group, or permissions, use the commands 
chown, chgrp, and chmod, respectively, as described in “Proper- 
ties of Files” on page 83. 


Selected Features of Bash 


A shell does much more than simply run commands. It also 
simplifies the running of commands, thanks to powerful fea- 
tures: pattern matching for filenames, a “command history” to 
recall previous commands quickly, pipes to send the output of 
one command to the input of another, variables to store values 
for use by the shell, and more. Take the time to learn these 
features, and you will become faster and more productive with 
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Linux.’ Let's skim the surface and introduce you to these useful 
tools. (For full documentation, run info bash.) 


Which Shell Are You Running? 


This book assumes that your shell is bash. To identify your 
shell, run: 


> echo SSHELL 
/bin/bash 


If your shell isn’t bash and you wish to try it, run the command 
bash directly, because bash, like all shells, is just a program. (It’s 
located at /bin/bash.) 


> bash 


Run the command exit when done to return to your regular 
shell. To change your default shell to bash, see the chsh com- 
mand in “User Account Management” on page 188. 


Pattern Matching 


Pattern matching in the shell, sometimes called wildcards, is a 
shorthand to work with sets of files. For example, the pattern a* 
refers to files whose names begin with lowercase “a.” The shell 
expands a pattern into the full set of filenames it matches. If 


you run: 


> 1s a* 
aardvark adamantium apple 


the shell invisibly expands the pattern a* into the filenames that 


begin with “a” in your current directory, as if you had typed: 
> ls aardvark adamantium apple 


ls never knows you used a pattern: it sees only the final list 
of filenames after expansion. This means every Linux program 


7 Also see my follow-up book, Efficient Linux at the Command Line, to 
grow your skills. 
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you launch from a shell, regardless of its origin, “works” with 
patterns and other shell features. This is a critically important 
point. A surprising number of Linux users think that programs 
expand their own file patterns on the command line. They 
don't. The shell does it before the associated program even runs. 


Patterns never match two special characters: a leading period 
and the directory slash (/). These characters must be given 
literally. A pattern like .bas* matches .bashrc, and /etc/*conf 
matches all filenames ending in conf in the /etc directory. 


Dot Files 


Filenames with a leading period, called dot files, are often hid- 
den from view unless you explicitly request them. An example 
is the bash initialization file .bashrc in your home directory. 


* ls omits dot files from directory listings, unless you pro- 
vide the -a option. 


e Pattern-matching characters in the shell do not match a 
leading period. 


Asa result, dot files are often called “hidden files.” 


Pattern Meaning 


* Zero or more consecutive characters, except a leading dot or a 
directory slash. 


? Any single character, except a leading dot or a directory slash. 


[set] Any single character in the given set. It can be a sequence 
of characters, like [ aeit ouAEIOU] for all vowels, or a 
range with a dash, like [A-Z] for all capital letters, ora 
combination. 


[ !set] Any single character not in the given set, such as [ !0-9] to 
mean any nondigit. 


[set] Same as [ ! set]. 
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To match a literal dash in a character set, put it first or last 
so it’s not part of a range. To include a literal closing square 
bracket in the set, put it first in the set, or escape it with a 
backslash (\]). To include a * or ! symbol literally, place it 
somewhere other than first in the set, or escape it. 


Brace Expansion 


Similar to file patterns, expressions with curly braces also 
expand to become multiple arguments to a command. The 
comma-separated expression: 


{bubble, quick ,merge} 


expands first to bubble, then quick, and finally merge within a 
command line, like this: 


> echo {bubble, quick,merge}sort. java 
bubblesort. java quicksort. java mergesort. java 


NOTE 


The key difference between braces and square brackets is 
that braces work with any strings, whereas square bracket 
expressions match only existing filenames. 


Curly braces can also expand to a sequence of values in a range, 
if you separate the endpoints of the range with two dots (. .): 


echo {3..12} 

456789 10 11 12 

echo {A..E} 

BCDE 

echo file{1..5}.py 

filel.py file2.py file3.py file4.py fileS.py 


Shell Variables 


You can define variables in a shell and assign them values: 


vQRPovwey 
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> MYVAR=3 Assign the value 3 to variable MYVAR 


To refer to a value, simply place a dollar sign in front of the 
variable name: 


> echo SMYVAR 
3 


The shell defines some standard variables when you log in: 


Variable Meaning 


DISPLAY The name of your X window display 

HOME Your home directory, such as /home/smith 

LOGNAME Your login name, such as smith 

MAIL Your incoming mailbox, such as /var/spool/mail/smith 
OLDPWD Your shell’s previous directory, prior to the last cd command 
PATH Your shell search path: directories separated by colons 

PWD Your shell’s current directory 

SHELL The path to your shell (¢.g., /bin/bash) 

TERM The type of your terminal (e.g., xterm or vt100) 

USER Your login name 


Variables and their values are limited, by default, to the shell 
that defines them. To make a variable and its value available to 
other programs your shell invokes (i.e., subprocesses), use the 
export command: 


> MYVAR=3 
> export MYVAR 


or the shorthand: 
> export MYVAR=3 


Your exported variable is now called an environment variable. 
To go further and make a variable available to every new shell 
you run, not just subprocesses of your current shell, place the 
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variable definition beforehand in a shell configuration file; see 
“Tailoring Shell Behavior” on page 58. 


To list a shell’s environment variables, run: 
> printenv 


To set an environment variable just for the duration of one 
command, prepend variable=value to the command line: 


> printenv HOME 


/home/smith 

> HOME=/home/sally printenv HOME 

/home/sally 

> printenv HOME 

/home/smith The original value is unaffected 


Search Path 


Programs may be scattered all over the Linux filesystem, mostly 
in directories like /bin and /usr/bin. When you run a command 
that invokes a program, somehow the shell must locate the 
program in the filesystem: 


> who The shell must locate the “who” program to run it 


The shell finds the program by consulting the value of the envi- 
ronment variable PATH, which is a list of directories separated 
by colons. This list is called the shell’s search path. 


> echo SPATH 
/usr/local/bin: /bin: /usr/bin Search path with 3 directories 


The shell looks for an executable file named who in each listed 
directory in sequence. If it locates who (say, in /usr/bin/who), 
it executes the program, and also caches the location for next 
time (run hash --help for more on caching). Otherwise, it 
reports a failure: 


bash: who: command not found 


To print a command's location in your search path, run the 
type or which command: 
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> type who 

who is /usr/bin/who 
> which who 
/usr/bin/who 


To add directories to your shell’s search path temporarily, mod- 
ify its PATH variable. For example, append /usr/sbin to your 
shell’s search path: 


> PATH=SPATH: /usr/sbin 
> echo $PATH 
/usr/local/bin:/bin: /usr/bin: /usr/sbin 


This change affects only the current shell. To make it stick, 
modify PATH in a bash configuration file, as explained in “Tai- 
loring Shell Behavior” on page 58. Then log out and log back 
in, or run the configuration file by hand in each open shell 
window. For example: 


> . S$HOME/.bashrc If you modified $HOME/.bashre 


Aliases 


The alias command defines a convenient shorthand for 
another command. For example, this alias: 


> alias Lll='ls -1G' 


defines anew command 11 that runs ls -1G: 


-=u 
total 436 
-fW-P--r-- 1 smith 3584 Oct 11 14:59 filet 


-Pwxr-xr-x 1 smith 72 Aug 6 23:04 file2 


Define aliases in your ~/.bashrc file (see “Tailoring Shell Behav- 
ior” on page 58) to make them available to future shells.* To 
list all your aliases, run alias. If you want more flexibility than 


8 Some setups use ~/.bash_aliases for this purpose. 
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aliases provide, see “Programming with Shell Scripts” on page 
311, run info bash, and read up on “shell functions” 


Built-in Commands 


Most Linux commands are programs in the Linux filesystem. 
Examples are wc and who, which usually live in the direc- 
tory /usr/bin. The shell locates and runs them using the PATH 
variable, as I described in “Search Path” on page 41. Some other 
commands, however, are built-in features of the shell, known 
as built-in commands. You've seen several built-in commands 
in this chapter, such as cd, alias, and export. To determine 
whether a command is in the filesystem, a built-in command, 
or an alias, run the type command: 


> type we cd 11 Print the types of these commands 
we is /usr/bin/we A program in the filesystem 
cd is a shell builtin A built-in shell command 


Ll is aliased to ‘ls -1G' An alias 


Input, Output, and Redirection 


Most Linux commands accept input and/or produce output. 
Keyboard input is called standard input or stdin. Output to 
your display is called standard output or stdout. Error messages 
are treated specially and printed on standard error or stderr, 
which also is usually your display, but Linux separates stderr 
from stdout internally.’ 


The shell can redirect standard input, standard output, and 
standard error to and from files. In other words, any command 
that reads from standard input can have its input come from a 
file instead with the shell’s < operator: 


> command < infile 


9 For example, you can capture standard output in a file and still have 
standard error messages appear on screen. 
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Likewise, any command that writes to standard output can 
write to a file instead: 


> command > outfile Create/overwrite outfile 
> command >> outfile Append to outfile 


A command that writes to standard error can have its error 
output redirected to a file as well, leaving standard output 
unaffected: 


> command 2> errorfile 
To redirect both standard output and standard error to files: 


> command > outfile 2> errorfile Separate files 
> command &> outfile Single file (preferred) 
> command >& outfile Single file (less common) 


Combined Commands 

Bash lets you move beyond simple commands by combining 
multiple programs on a single command line. 

Sequences of commands 


To invoke several commands in sequence on a single command 
line, separate them with semicolons: 


> command1 ; command2 ; command3 


To run a sequence of commands as before, but stop execution if 
any of them fails, separate them with && (“and”) symbols: 


> command1 && command2 && command3 


To run a sequence of commands, stopping execution as soon as 
one succeeds, separate them with || (“or”) symbols: 


> command1 || command2 || command3 
Pipes 


You can redirect the standard output of one command to be 
the standard input of another, using the shell’s pipe (|) operator. 
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(On US keyboards, find this symbol just above the Enter key.) 
For example, this command: 


> who | sort 


sends the output of who into the sort command, printing an 
alphabetically sorted list of logged-in users. Multiple pipes 
work too. Let’s sort the output of who again, extract the first 
column of information (using awk), and display the results one 
page at a time (using less): 


> who | sort | awk '{print $1}' | less 


Command substitution 


If you surround a command with backquotes (“backticks”), the 
shell removes the command and substitutes the command’s 
output. 


> date +%Y Print the current year 
2024 

> echo This year is “date +%Y~ 

This year is 2024 


A dollar sign and parentheses are equivalent to backquotes: 


> echo This year is $(date +%Y) 
This year is 2024 


but are superior because they can be nested: 


> echo Next year is $(expr $(date +%Y) + 1) 
Next year is 2025 


Process substitution 


Some programs don’t work well with pipes because they don't 
read from standard input, only from disk files. An example 
is the diff command that compares two files line by line 
and prints their differences. Process substitution is a way to 
force a command like diff to read from standard input. It 
runs a command and lets its output “masquerade” as a file, 
which programs like diff will happily accept. With the process 
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substitution operator, <(), you can compare the output of two 
commands instead of two disk files. 


Suppose you have a directory full of JPEG and text files in 
pairs: 
> 1s jpegexample 


file1.jpg file2.jpg file3.jpg 
file1.txt file2.txt file3.txt 


and you want to confirm that every JPEG file has a correspond- 
ing text file and vice versa. Ordinarily, you might create two 
temporary files, one containing the JPEG filenames and the 
other containing the text filenames, remove the file extensions 
with cut, and compare the two temporary files with diff: 


> cd jpegexample 

> ls *.jpg | cut -d. -f1 > /tmp/jpegs 
> ls *.txt | cut -d. -f1 > /tmp/texts 
> diff /tmp/jpegs /tmp/texts 


5a6 
> file6 No file6.jpg was found 
8d8 
< filed No file9.txt was found 


Process substitution performs the same task with a single com- 
mand and no temporary files: 


> diff <(ls *.jpg|cut -d. -f1) <(ls *.txt|cut -d. -f1) 


Each <() operator stands in for a filename on the command 
line, as if that “file” contained the output of ls and cut. 


Preventing Evaluation 

The shell evaluates every character of a command. To prevent 
evaluation, use quoting or escaping. 

Quoting 


Normally, the shell treats whitespace as a separator for strings 
on the command line. To make a string that contains white- 
space (e.g., a filename with a space in it), surround it with 
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single or double quotes, and the shell treats it as a unit. Single 
quotes treat their contents literally, while double quotes permit 
variables and other shell constructs to be evaluated: 


> echo 'The variable HOME has value SHOME' 
The variable HOME has value SHOME 

> echo "The variable HOME has value SHOME" 
The variable HOME has value /home/smith 


Escaping 


If a character has special meaning to the shell but you want 
it used literally (e.g., * as a literal asterisk rather than a file 
pattern), precede the character with the backslash “\” character. 
This is called escaping the special character: 


> echo a* A file pattern 
aardvark adamantium apple 

> echo a\* A literal asterisk 

a* 

> echo "I live in SHOME" Print a variable value 

I live in /home/smith 

> echo "I live in \$HOME" Print a literal dollar sign 


I live in SHOME 


You can also escape control characters (tabs, newlines, ‘D, etc.) 
to have them used literally on the command line, if you precede 
them with *v. This is particularly useful for tab characters, 
which the shell would otherwise use for filename completion 
(see “Filename Completion” on page 49). 


> echo "There is a tab between here4V and here" 
There is a tab between here and here 


Command-line Editing 


Bash lets you edit the command line youre working on, 
using keystrokes inspired by the text editors Emacs and Vim 
(see “Creating and Editing Files” on page 78). To enable 
command-line editing with Emacs keys, run this command 
(and place it in a bash configuration file to make it permanent): 
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> set -o emacs 
For vi (or Vim) keys: 


> set -o vi 


Emacs keystroke Vim keystroke Meaning 
(after ESC) 
AP or up arrow k or up arrow Go to previous command 
AN or down arrow jordownarrow Go to next command 
AR Search for a previous 
command interactively 
AF or right arrow Lor right arrow Go forward one character 
AB or left arrow hor left arrow Go backward one character 
AA 0 Go to beginning of line 
AE $ Go to end of line 
AD x Delete next character 
AU AU Delete to beginning of line 


Command History 


A shell can recall previous commands and re-execute them, a 
feature called command history. Try these useful history-related 
commands and expressions: 


Command Meaning 


history Print your history 
history N Print the most recent V commands in your history 
history -c Clear (delete) your history 


im Represents your previous command. To re-run it: 
> !! <Enter> 


IN Represents command number Win your history 


!-N Represents the command you entered NW’ commands ago 
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Command Meaning 


1§ Represents the last argument from the previous 
command. Great for checking that files are present before 
running a destructive command like rm: 


> ls z* 
zebra.txt zipfile.zip zookeeper 
> rm !$ Same as “rm z*” 
I* Represents all arguments from the previous command: 


> 1s myfile emptyfile hugefile 
emptyfile hugefile myfile 
* we {* 
18 211 1168 myfile 
0 0 0 emptyfile 
333563 2737540 18577839 hugefile 
333581 2737751 18579007 total 


Filename Completion 


In the middle of typing a filename, press the Tab key and the 
shell automatically completes the filename for you. If several 
filenames match what you've entered so far, the shell beeps, 
indicating the match is ambiguous. Immediately press Tab a 
second time and the shell presents the alternatives. Try this: 


> cd /usr/bin 
> ls un<Tab><Tab> 


The shell displays all files in /usr/bin that begin with un, such 
as uniq and unzip. Enter a few more characters to disambiguate 
your choice and press Tab again. 


Shell Job Control 


jobs List your jobs. 
& Placed after a command, runs it in the background. 


AZ Keystroke to suspend the current (foreground) job. 
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suspend Suspenda shell. 
fg Unsuspend a job: bring it into the foreground. 
bg Make a suspended job run in the background. 


disown — Forgetajob. 


All Linux shells have job control: the ability to run commands 
in the background (multitasking behind the scenes) and fore- 
ground (the active process at your shell prompt). A job is 
simply the shell’s unit of work. When you run a command 
interactively, your current shell tracks it as a job. When the 
command completes, the associated job disappears. Jobs are 
at a higher level than Linux processes; the Linux OS knows 
nothing about them. They are merely constructs of the shell. 
Here is some important vocabulary about job control: 


Foreground job 
In a shell, a running job that occupies the shell prompt so 
you cannot run another command 


Background job 
In a shell, a running job that doesn’t occupy the prompt, 
so you can run other commands in the same shell 


Suspend 
To stop a foreground job temporarily 


Resume 
To cause a suspended job to start running in the fore- 
ground again 


Disown 
To tell the shell to stop tracking the job; the underlying 
processes continue to run 
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jobs stdout ~-help 


The built-in command jobs lists the jobs running in your 
current shell by number and name: 


> jobs 
[1]- Running emacs myfile & A background job 
[2]+ Stopped ssh example.com A suspended job 


The integer on the left is the job number, and the plus sign 
identifies the default job affected by the fg (foreground) and bg 
(background) commands. 


& 


Placed at the end of a command line, the ampersand causes the 
given command to run as a background job: 


> emacs myfile & 
[2] 28090 


The shell’s response includes the job number (2) and the pro- 
cess ID of the command (28090). 


AL 


Typing *Z in a shell, while a job runs in the foreground, 
suspends that job. It simply stops running, but its state is 
remembered: 


> sleep 10 Waits for 10 seconds 
cad 
[1]+ Stopped sleep 10 


> 


Selected Features of Bash | 51 


Now you're ready to run bg to put the sleep command into the 
background, or fg to resume it in the foreground. You could 
also leave it suspended and run other commands. 


suspend ~help 


The built-in suspend command pauses the current shell if pos- 
sible, as if youd applied *Z to the shell itself. For instance, if you 
create a superuser shell with sudo and want to return to your 
original shell, suspend pauses the superuser shell: 


> whoami 

smith 

> sudo bash Run a superuser shell 
[sudo] password: xxxXxXxxxx 

# whoami 

root 

# suspend Suspend the superuser shell 


[1]+ Stopped sudo bash 
> whoami Back to the original shell 
smith 


bg ~help 
bg [%job] 


The built-in command bg sends a suspended job to run in 
the background. With no arguments, bg operates on the most 
recently suspended job. To specify a particular job (shown by 
the jobs command), supply the job number or name preceded 
by a percent sign: 


> bg %2 Send job 2 to the background 
> bg %cat Send job beginning with “cat” to the background 
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Some types of interactive jobs cannot remain in the back- 
ground—for instance, if they are waiting for input. If you try, 
the shell suspends the job and displays: 


[2]+ Stopped command line here 


Now resume the job (with fg) and continue. 


fg --help 
fg [%job] 


The built-in command fg brings a suspended or backgrounded 
job into the foreground. With no arguments, it selects a job, 
usually the most recently suspended or backgrounded one. 
To specify a particular job (as shown by the jobs command), 
supply the job number or name preceded by a percent sign: 


> fg %2 Bring job 2 into the foreground 
> fg %cat Bring job beginning with “cat” into the foreground 


disown “ng 
disown [-ar] [-h] [%job] 


The built-in command disown tells your current shell to “for- 
get” a job. The Linux processes behind the job keep running— 
you just can’t control them anymore with bg, fg, jobs, and 
other job-related commands. This is useful for long jobs that 
you don't need to interact with, or jobs that should keep run- 
ning after your shell exits. See also nohup in “Controlling Pro- 
cesses” on page 169. 


> disown %2 Forget job #2 

> disown %cat Forget job beginning with “cat” 

> disown -h %2 Mark job #2 to keep running after shell exits 
> disown -r Forget all running jobs 

> disown -a Forget all jobs 
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Running Multiple Shells at Once 


Job control can manage several commands at once, but only 
one can run in the foreground at a time. More powerfully, 
you can run multiple shells at once, each with a foreground 
command and any number of background commands. 


If your Linux computer runs a window system such as KDE 
or GNOME, you can easily run many shells at the same time 
by opening multiple shell windows (see “Launching a Shell” on 
page 21). In addition, certain shell window programs, such as 
KDE's konsole, can open multiple tabs within a single window, 
each one running a shell. 


Even without a window system—say, over an SSH network 
connection—you can manage multiple shells at once. The tmux 
command simulates multiple shell windows in an ordinary 
ASCII terminal. Using special keystrokes, you can switch from 
one virtual window to another at will. (Another such program 
is screen, but tmux is better maintained and easier to config- 
ure.) To begin a session with tmux, run: 


> tmux 


A new shell launches with an extra status bar at the bottom 
of the terminal, indicating that you’re running one virtual win- 
dow. The tmux program provides 10 such windows by default, 
labeled from 0 to 9, that you may switch between. Each window 
runs a single shell at first, but you can split a window into 
multiple “panes” to display multiple shells at once. Try these 
keystrokes to get the hang of tmux: 


1. In the current tmux window, run ls. 


2. Press *Bc (Ctrl-B, then press c). tmux displays a fresh 
shell prompt in a second virtual window. The status bar 
changes to show two virtual windows numbered 0 and 1. 


3. In this second window, run a different command (say, df). 


4. Press “Bn and you'll switch back to window 0, where your 
output from 1s is now visible again. 
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5. Press “Bn a few more times to toggle between the two 
virtual windows. 


6. Press *B% to split the current window into two panes side 
by side. 


7. Press *B" to split the current pane into two, vertically. 
You're now viewing three shells in separate panes. 


Most aspects of tmux are configurable in the file ~/.tmux_conf, 
even the choice of *B as the prefix key. Here are common 
keystroke commands: 


AB? Display online help. Press “q’ to quit. 

ABC Create a window. 

ABO, *B1...*B9 Switch to window 0 through 9, respectively. 

ABn Switch to the next window, numerically. 

ABp Switch to the previous window, numerically. 

ABL Switch to the most recently used window. 

AB% Split into two panes side by side. 

AB" Split into two panes top and bottom. 

“Bo Jump to the next pane. 

AB left arrow Jump to the pane to the left. 

AB right arrow Jump to the pane to the right. 

AB up arrow Jump to the pane above. 

“B down arrow Jump to the pane below. 

ABq Display pane numbers for reference. 

ABX Kill the current pane. 

ABAB Send a true Ctrl-B to your shell, ignored by 
tmux. 

ABAZ Suspend tmux. 
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Keystroke Meaning 


ABd “Detach” from a tmux session and return to 


your original shell. To return to tmux, run 
tmux attach. 


a) Terminate a shell in a window or pane. 


This is the ordinary “end of file” keystroke, 
explained in “Terminating a Shell” on page 
58, which closes any shell. 


4B: kill-session Kill all windows and terminate tmux. 


A few notes about running tmux: 


e If shells within tmux are missing your aliases, variables, 
or other shell settings, that’s because tmux runs a login 
shell that does not source your .bashrc initialization file. 
It only sources your startup file (.bash_profile, .bash_login, 
or .profile). To correct this issue, append these lines to 
your startup file: 

# Source my .bashrc file 

if [ -f ~/.bashrc ]; then 
. ~/.bashre 

fi 

¢ Ifyou run a text editor, tmux captures all Ctrl-B keystrokes, 
even those intended as editing commands. Press *B‘B to 
send a true Ctrl-B to your editor. 


e Dont run tmux locally on a graphical desktop; run multiple 
shell windows instead. It’s easier and avoids a problem: if 
you've configured your shell to run commands on logout 
(for example, in the file ~/.bash_logout), tmux’s shells will 
run those commands on exit, even though you haven't 
logged out of the desktop. This may have unwanted effects 
on your desktop login session. 
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Killing a Command in Progress 


To kill a foreground command immediately, press *c. Here I 
kill the cat command as it prints a huge file: 


> cat hugefile 

Lorem ipsum dolor sit amet, consectetur adipiscing 
odio. Praesent libero. Sed cursus ante dapibus diam. 
quis sem at nibh elementum blah blah blah 4C 


> 


To kill a background command, bring it into the foreground 
with fg and then press “C: 


> sleep 50 & 

[1] 12752 

> jobs 

[1]- Running sleep 50 & 
> fg %1 

sleep 50 

4C 


or run the kill command described in “Controlling Processes” 
on page 169. The keystroke “Cc is a shell feature. It has no effect 


on programs that “catch” *c and do not terminate, like text 
editors and GUI applications. Use kill for those. 


Surviving a Kill 


Killing a command with *c may leave your shell in an odd 
or unresponsive state, because the killed program could not 
close itself properly. A common symptom is not displaying the 
keystrokes you type. To fix the shell, follow these steps: 


1. Press “J to get a shell prompt. This keystroke may work 
even if pressing Enter does not. 


2. Type the command reset (even if the letters don’t appear 
while you type) and press “J to run it. Your shell should 
return to normal. 
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Terminating a Shell 
To terminate a shell, either run the exit command: 
> exit 


or press ‘D on a line by itself. The keystroke “D sends an “end of 
file” signal to any program reading from standard input. This 
includes the shell itself. 


Tailoring Shell Behavior 


Several files in your home directory control the behavior of 
bash shells. The startup files .bash_profile, .bash_login, and .pro- 
file contain commands that run each time you log in. (Choose 
just one startup file and stick with it. I recommend .bash_pro- 
file because some other shells also use .profile.) Commands 
in the initialization file .bashrc run every time you launch an 
interactive shell, and commands in .bash_logout run each time 
you log out. All these files can set variables, run programs, 
print silly messages, or whatever you like. Other Linux shells 
use other configuration files as shown in Table 1-1. 


Table 1-1. Shell configuration files in $HOME and when they are read 


On login By other interactive shells On logout 
bash .bash_profile, .bash_login, .bashrc .bash_logout 
profile 


dash _.profile 
fish .config/fish/config.fish .config/fish/config.fish 


ksh? .profile, .kshrc skshrc 
tesh _ .tcshrc, .cshrc, .login .teshre, .cshrc 
zsh? _.zshenv, .zprofile, .zlogin .Zshenv, .zshrc .Zlogout 


“To override these file paths with environment variables, see the manpage. 


Other shell configuration files live in /etc for system-wide con- 
trol; see the respective manual page, or manpage, for each 
shell. All these configuration files are examples of shell scripts: 
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executable files that contain shell commands. I cover this fea- 
ture in more detail in “Programming with Shell Scripts” on 
page 311. 


Getting Help 


If you need more information than this book provides, here are 
several ways to get more help: 


Run the man command 
The man command displays documentation for a given 
program. For example, to learn about counting words in a 
file with we, run: 


> man wc 


To search for manual pages (manpages) by keyword for 
a particular topic, use the -k option followed by the 
keyword: 


> man -k database 


If the list of manpages is longer than the screen, pipe it to 
less to display it in pages (press q to quit): 


> man -k database | less 


Run the info command 
The info command is an extended, hypertext help system 
covering many Linux commands. 


> info ls 
While info runs, some useful keystrokes are: 
¢ To get help, press h 
e To quit, press q 


e To page forward and backward, use the space bar and 
Backspace key, respectively 


¢ To jump between hyperlinks, press Tab 
e To follow a hyperlink, press Enter 
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If info has no documentation on a given command, 
it displays the command’s manpage. For a listing of 
available documentation, type info by itself. To learn 
how to navigate the info system, run info info. 


Use the --help option (if any) 
Many Linux commands respond to the option --help or 
-h by printing a short help message. Try: 


> we --help 
If the output is longer than the screen, pipe it into less: 
> we --help | less 


Examine the directory /usr/share/doc 
This directory contains supporting documents for pro- 
grams, usually organized by name and version. For exam- 
ple, files for the editor Emacs version 28 are likely found 
(depending on distro) in /usr/share/doc/emacs28. 


Distro-specific websites 
Most Linux distros have an official site with documenta- 
tion, discussion forums, and other resources. Search the 
web for your distro name (e.g., “Ubuntu”) to find its 
website. The Arch Linux wiki is particularly informative 
regardless of your distro. 


Linux help sites 
Ask Linux questions at cunix.stakexchange.org, linuxques- 
tions.org, itsfoss.community, and nixcraft.com. 


Web search 
To decipher a Linux error message, paste it into a search 
engine, verbatim, optionally surrounded by double quotes. 


This concludes my basic overview of Linux and the shell. Now 
let’s turn to the specifics of Linux commands. The rest of the 
book lists and describes the most useful commands to work 
with files, processes, users, networking, multimedia, and more. 
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CHAPTER 2 
File Commands 


Basic File Operations 


ls __ List files ina directory. 

cp Copyafile. 

mv Move (rename) a file. 

rm Remove (delete) a file. 

Ln Create links (alternative names) to a file. 


One of the first things you'll do on a Linux system is manipu- 
late files: copying, renaming, deleting, and so forth. 


Is stdout --opt --help — --version 
ls [options] [files] 


The ls command (pronounced as it is spelled, ell ess) lists 
attributes of files and directories. You can list files in the cur- 
rent directory: 


> 1s 
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in given directories: 


> 1s diri dir2 dir3 


or individually: 


> 1s myfile myfile2 myfile3 


The most important ls options are -a, -1, and -d. By default, 
ls hides files whose names begin with a dot, as explained in 
the sidebar “Dot Files” on page 38. The -a option displays all 
files. Depending on your account settings, 1s will list dot files at 
the beginning (sorting based on the dot) or mix them into the 


TIP 


If ls behaves differently than you expect, your distro may 
have defined an alias for 1s (see “Aliases” on page 42). 
Check for an alias by running this command: 


> alias ls 

alias ls='/bin/ls -FHN' Yes, there's an alias 
To run the original command rather than the alias, pre- 
pend a backslash (\ls). To remove the alias, run unalias 
ls. Then look in your shell configuration files for the alias 
definition (see “Tailoring Shell Behavior” on page 58) and 
remove it. If you don’t see a definition, either add the 
command unalias ls or define a new alias that works the 
way you want, like alias ls="/bin/ls". 


listing (sorting based on the character after the dot). 


> ls 

myfile myfile2 

> ls -a 

-hidden_file myfile  myfile2 
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The -1 option produces a long listing: 


> 1s -l myfile 
-rw-r--r-- 1 smith users 1168 Oct 28 2015 myfile 


that includes, from left to right: the file’s permissions (-rw-r-- 
r--), number of hard links (1), owner (smith), group (users), 
size (1168 bytes), last modification date (Oct 28 2015) and 
name. See “File Permissions” on page 35 for more information 
on these attributes. 


The -d option lists information about a directory itself, rather 
than descending into the directory to list its files: 


> 1s -ld diri 
drwxr-xr-x 1 smith users 4096 Oct 29 2015 dir1 


Useful options 


List all files, including those whose names begin with a dot. 


Long listing, including file attributes. Add the -h option (human-readable) 
to print file sizes in KB, MB, and GB, instead of bytes. 


Ina long listing, print file sizes in friendly KB, MB, and other human- 
readable terms, instead of bytes. 


Ina long listing, don’t print the group ownership of the file. 


Decorate filenames with meaningful symbols, indicating their types. 
Appends “/” to directories, “*” to executables, “@” to symbolic links, 
to named pipes, and “=" to sockets. These are just visual indicators, not part 
of the filenames! 


lp 


Sort files by their size. 

Sort files by the time they were last modified. 
Reverse the sorted order. 

If listing a directory, list its contents recursively. 


If listing a directory, do not list its contents, just the directory itself. 
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cp --opt --help — --version 


cp [options] source_file destination_file 
cp [options] (files | directories) directory 
The cp command copies one file to another: 
> cp myfile anotherfile 
or copies multiple files into a directory (say) mydir: 
> cp myfile myfile2 myfile3 mydir 


Use the -a or -r option to copy directories recursively, includ- 
ing all subdirectories and their contents. For more sophistica- 
ted copying, see rsync in “Backups and Remote Storage” on 
page 241. 


Useful options 


-p Copy not only the file contents, but also the file’s permissions, timestamps, 
and, if you have sufficient permission to do so, its owner and group. 
(Otherwise, you own the copies, the timestamp is now, and permissions are 
set by applying your umask to the original permissions.) 


-a Copy a directory hierarchy recursively, preserving all file attributes and links. 


-r Copy a directory hierarchy recursively. This option does not preserve the files’ 
attributes such as permissions and timestamps. It does preserve symbolic 
links. 


-i_ Interactive mode. Ask before overwriting destination files. 


-f Force the copy. Ifa destination file exists, overwrite it unconditionally. 


mv --opt --help — --version 
mv [options] sources target 


The mv (move) command renames a file: 


> mv somefile yetanotherfile 
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or moves files and directories into a destination directory: 
> mv myfile myfile2 diri dir2 destination_directory 
Useful options 


-i__ Interactive mode. Ask before overwriting destination files. 


-f Force the move. Ifa destination file exists, overwrite it unconditionally. 


rm --opt --help — --version 


rm [options] files | directories 

The rm (remove) command deletes files: 
> rm deleteme deleteme2 

or recursively deletes directories: 


> rm -r diri1 dir2 


WARNING 


Use rm -r with caution. It can quickly destroy large num- 
bers of files, especially when combined with -f to suppress 
prompts and ignore errors. 


Use sudo rm -r with extreme caution. It can destroy your 
whole operating system. 


Useful options 


-i_ Interactive mode. Ask for confirmation before deleting each file. 
-f Force the deletion, ignoring any errors or warnings. 


-r Recursively remove a directory and its contents. 


Basic File Operations 
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In --opt --help — --version 
ln [options] source target 


The ln command creates a link, which lets a file live at more 
than one location in the filesystem at once. There are two kinds 
of links, illustrated in Figure 2-1. A hard link is a second name 
for the same physical file on disk. In tech jargon, both refer to 
the same inode, a data structure that locates a file's content on 
disk. The following command creates a hard link, myhardlink, 
to the file named myfile: 


> 1n myfile myhardlink 


Make a file: 


§ touch name_A 


name_A sannueeaocaaees| 
(ordinary file) Location 
of file 
on disk 
Hard link: 
$ In name_A name_B 
Marne A sassessseecses 
Location 
of file 
MAME _B nsssseeseeeees on disk 
Symbolic link: 


S$ ln -s name_A name_B 


MAME _A evssssseeeeeeee 
* Location 
: of file 
name_B on disk 
(Symbolic link) 


Figure 2-1. A hard link points to a file’s data. A symbolic link points to 
a file’s path. 
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A symbolic link (also called a symlink or soft link) is a pointer 
to the path (not the inode) of another file or directory. If you're 
familiar with Windows shortcuts or macOS aliases that point 
to another file or folder, they are much like symbolic links. To 
create a symbolic link, add the -s option: 


> 1n -s myfile mysoftlink 


Hard and symbolic links have important differences: 


¢ If you rename or delete the original file, a hard link is not 
affected: it still points to the same data that the original file 
did. A symbolic link breaks, however. It still points to the 
original file's path, which no longer exists. The symbolic 
link is now “dangling,” and if you use it in a command, 
youll probably receive a “file not found” error. 


e Hard links can exist only on the same device as the orig- 
inal file because inodes have meaning only on a single 
device. Symbolic links can point to files on other devices 
because they refer to file paths, not file data. 


Symbolic links can point to directories, whereas hard links 
generally cannot. (On some filesystems, the superuser can 
create a hard link to a directory with the -d option.) 


Useful options 


-s Make asymboliclink instead of a hard link. 
-i__ Interactive mode. Ask before overwriting target files. 
-f Force the link. If the target already exists, overwrite it unconditionally. 


-b Create a backup. If the target already exists, rename it by appending a tilde, 
then create the link. 


-d Create a hard link to a directory if possible (superusers only). 
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To find out where a symbolic link points, run either of the 
following commands, which show that the link examplelink 
points to the file myfile: 


> readlink examplelink 

myfile 

> 1s -l examplelink 

Lrwxrwxrwx 1 smith aes examplelink -> myfile 


Symbolic links can point to other symbolic links. To follow an 
entire chain of links to discover where they point in the end, 
use readlink -f. 


Directory Operations 


cd Change your current directory. 
pwd Print the name of your current directory. 
basename Print the final part of a file path, usually the filename. 


dirname _ Printa file path without its final part. 


mkdir Create (make) a directory. 
rmdir Delete (remove) an empty directory. 
rm -r Delete a nonempty directory and its contents. 


I discussed the directory structure of Linux in “The Filesystem” 
on page 27. Let’s cover commands that create, modify, delete, 
and manipulate directories within that structure. 


cd stdout --help 
cd [directory] 


The cd (change directory) command sets the current directory 
of your shell: 


> cd /usr/games 
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With no directory supplied, cd defaults to your home directory: 
> cd 


If the supplied directory is a dash (-), cd returns to the previous 
directory it visited in the current shell and prints its path. 


> cd /etc Start in /etc 

> cd /bin Go somewhere else 

> cd - Return to /etc 

/etc 
pwd stdout --help 
pwd 


The pwd command (“print working directory”) prints the abso- 
lute path of your current directory: 


> pwd 
/users/smith/linuxpocketguide 


basename stdout help --version 
basename path [extension] 


The basename command prints the final part of a file path. The 
path doesn’t have to exist in your filesystem. 


> basename /users/smith/finances/money.txt 
money. txt 

> basename any/string/you/want Arbitrary string 
want 


If you provide an optional extension, it is stripped from the 
result: 


> basename /users/smith/finances/money.txt .txt 
money 


Directory Operations | 69 


dirname stdout -help — --version 
dirname path 


The dirname command prints a file path with its final part 
removed: 


> dirname /users/smith/mydir 
/users/smith 


dirname does not change your shell’s current directory. It 
manipulates and prints a string, just like basename does. 


mkdir --opt --help — --version 
mkdir [options] directories 


mkdir creates one or more directories: 


> mkdir directory1 directory2 directory3 


Useful options 


-p Given a directory path (not just a simple directory name), create any 
necessary parent directories automatically. The command: 
> mkdir -p one/two/three 


creates directories one and one/two and one/two/three if they don’t 
already exist. 


-m mode Create the directory with the given permissions: 
> mkdir -m 0755 publicdir 


By default, your shell’s umask controls the permissions. See the 
chmod command in “Properties of Files” on page 83, and “File 
Permissions” on page 35. 
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rmdir --opt --help — --version 
rmdir [options] directories 


The rmdir (remove directory) command deletes one or more 
empty directories given as arguments: 


> mkdir /tmp/junk create a directory 
> rmdir /tmp/junk delete it 


Useful options 


-p_ Ifyou supply a directory path (not just a simple directory name), delete not 
only the given directory, but the specified parent directories automatically, 
all of which must be empty. So rmdir -p one/two/three deletes not 
only one/two/three, but also one/two and one. 


To delete a nonempty directory and its contents, carefully run 
rm -r directory. Use rm -ri to delete interactively, or rm -rf to 
annihilate whole trees without error messages or confirmation. 


Viewing Files 


cat View files in their entirety. 

less View text files one page at a time. 

nl View text files with their lines numbered. 
head View the first lines of a text file. 

tail View the last lines of a text file. 


strings _ Display text that’s embedded in a binary file. 


od View data in octal (base 8) or other formats. 


Some files contain readable text, and others contain binary 
data. Let’s see how to display their contents in basic ways. 
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cat stdin stdout -file --opt --help  --version 
cat [options] [files] 


The simplest viewer is cat, which just prints its files to standard 
output, concatenating them (hence the name): 


> cat myfile Print one file on screen 
> cat myfile* Print many files 
> cat myfile* | we Concatenate files and pipe the text to we 


If a file contains more lines than your screen can display, use 
less to present the output one screenful at a time. 


Useful options 


-T Print tabs as “I. 

-E Print newlines as $. 

-v_ Print other nonprinting characters in a human-readable format. 

-n_ Prepend line numbers to every line. (The nl command is more powerful.) 
-b_ Prepend line numbers to nonblank lines. 


-s Squeeze consecutive blank lines into a single blank line. 


less stdin stdout -file --opt --help  --version 
less [options] [files] 


Use less to view text one “page” at a time (ie., one window or 
screenful at a time): 


> less myfile 


It’s great for long text files or as the final command in a shell 
pipeline with lengthy output: 


> command1 | command2 | command3 | command4 | less 
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While running less, press h for a help message describing all 
its features. Here are some useful keystrokes for paging: 


h,H View a help page. 

Space bar, f, *V,*F Move forward one screenful. 

Enter Move forward one line. 

b, 8B, ESC-v Move backward one screenful. 

/ Search mode. Type a regular expression and press Enter, 
and less locates the first matching line. 

? Same as /, but search backward. 

n Next match: Repeat your most recent search forward. 

N Repeat your most recent search backward. 

v Edit the current file with your default text editor (the 


value of environment variable VISUAL, or if not defined, 
EDITOR, or if not defined, the system default editor). 


<,g Jump to beginning of file. 
>,G Jump to end of file. 

mn Jump to next file. 

:p Jump to previous file. 


less has a mind-boggling number of features; I’m presenting 
only the most common. (For instance, in many distros, less 
can display the contents of a ZIP file: try less zipfile.zip.) 
The manpage is recommended reading. 


Useful options 


-c Clear the screen before displaying the next page. This avoids scrolling and 
may be more comfortable on the eyes. 


-m_ Print a verbose prompt showing the percentage of the file displayed so far. 


-N_ Display line numbers. 
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-r Display control characters literally; normally Less converts them toa 


human-readable format. 


-s Squeeze multiple, adjacent blank lines into a single blank line. 


-S Truncate long lines to the width of the screen, instead of wrapping. 


nl stdin stdout -file --opt --help 
nl [options] [files] 
nl prints files on standard output with line numbers: 


> nl poem 
1 Once upon a time, there was 
2 a little operating system named 
3 Linux, which everybody loved. 


nl provides more control over numbering than cat -n. 


Useful options 


--version 


-b [a]t|n]pR] Numberall lines (a), nonblank lines (t), no lines (n), or 
only lines that contain regular expression R. (Default=a) 


-v N Begin numbering with integer W. (Default=1) 


-iN Increment by N for each line. For example, print odd 
numbers only (-i2) or even numbers only (-v2 -i2). 


(Default=1) 


-n [ln|rn|rz] Format numbers as left-justified (Ln), right-justified (rn), 
or right-justified with leading zeros (rz). (Default=1n) 


-w N Force the width of the number to be N columns. 
(Default=6) 
-s S Insert string 5 between the line number and the text. 


(Default=Tab) 
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head stdin stdout -file --opt --help — --version 
head [options] [files] 


The head command prints the first 10 lines of a file, which is 
great to preview the contents: 


> head myfile 
> head myfile* | less = Previewing multiple files 


It’s also helpful for previewing output from a pipeline. List the 
10 most recently modified files in the current directory: 


> ls -lta | head 


Useful options 


-n N_ Print the first WV lines instead of 10. 
-N Same as -nN. 
-c N_ Print the first W bytes of the file. 


-q Quiet mode: when processing more than one file, don’t print a banner 
(containing the filename) above each file. 


tail stdin stdout -file --opt --help  --version 
tail [options] [files] 


The tail command prints the last 10 lines of a file: 


> tail myfile 
> nl myfile | tail See line numbers too 


The ultra-useful -f option causes tail to watch a file actively 
while another program writes to it, displaying new lines as they 
are written. This is invaluable for watching a Linux logfile that’s 
in active use: 


> tail -f /var/log/syslog Or another logfile 
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Useful options 


-n N _ Print the last W lines of the file instead of 10. 
-N Sameas-n N. 

-n +N Print from line NV to the end of the file. 

+N Same as -n +A. 

-c N Print the last NV bytes of the file. 


-f Keep the file open, and whenever lines are appended to the file, print 
them. Add the - - retry option if the file doesn’t exist yet. *C to quit. 


-q Quiet mode: when processing more than one file, don’t print a banner 
(containing the filename) above each file. 


strings stdin stdout -opt --help — --version 
strings [options] [files] 


Binary files, such as compiled programs, usually contain some 
readable text, like version information, authors’ names, and file 
paths. The strings command extracts that text: 


> strings /bin/bash 
/1ib64/1d-linux-x86-64.s0.2 

@(#)Bash version 5.1.16(1) release GNU 
comparison operator expected, found ‘%s’ 


Combine strings -n and grep to make your exploring more 
efficient. Let’s look for email addresses: 


> strings -n 10 /bin/bash | grep @ 
bash-maintainers@gnu.org 


Useful options 


-n length Display only strings with length greater than /ength (default=4). 


-f Prepend the filename to each line of output. 
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od stdin stdout -file --opt --help  --version 
od [options] [files] 


To view binary files, consider od (octal dump) for the job. It 
displays their data in ASCH, octal, decimal, hexadecimal, or 
floating point, in various sizes (byte, short, long). For example, 
this command: 


> od -w8 /usr/bin/who 

0000000 042577 043114 000402 000001 
0000010 000000 000000 000000 000000 
0000020 000003 000076 000001 000000 


displays the bytes in binary file /usr/bin/who in octal, eight 
bytes per line. The leftmost column is the file offset of each 
row, again in octal. 


If your binary file also contains text, consider the -tc option, 
which displays character data. For example, binary executables 
like who contain the string “ELF” at the beginning: 


> od -tc -w8 /usr/bin/who | head -3 

0000000 177 E L F Q02 001 001 \O 
9000010 \o \O \o \o \o \o \o \o 
9000020 003 \O > \0 001 \o \oO \o 


Useful options 


-NB Display the first B bytes of each file, specified in decimal, 
hexadecimal (by prepending Ox), 512-byte blocks (append 
b), KB (append k), or MB (append m). 


-jB Begin the output at byte B + 1 of each file; acceptable 
formats are the same as for the -N option. (Default=0) 


-w [B] Display B bytes per line; acceptable formats are the same 
as in the -N option. Using -w by itself is equivalent to 
-w32. (Default=16) 
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-s [B] Group each row of bytes into sequences of B bytes, 
separated by whitespace; acceptable formats are the same 
as in the -N option. Using - s by itself is equivalent to 
-s3. (Default=2) 


-A (djo|x|n) Display file offsets in the leftmost column, in decimal (d), 
octal (0), hexadecimal (x), or not at all (n). (Default=o) 


-t(alc)[z] Display output in a character format, with 
nonalphanumeric characters printed as escape sequences 
(c) or by name (a). 


-t(d|o|u|x)[z] Display output in an integer format: octal (0), signed 
decimal (d), unsigned decimal (u), hexadecimal (x). 


Appending z to the -t option displays the printable characters 
on each line in a column on the right side. 


Creating and Editing Files 


nano Asimple text editor found in virtually all Linux distros. 
emacs A powerful text editor from Free Software Foundation. 


vin A powerful text editor based on Unix vi. 


To get far with Linux, you must become proficient with one 
of its text editors. The three major ones are nano, Emacs, 
and Vim. Teaching these editors fully is beyond the scope 
of this book, but they all have online tutorials, and common 
operations are in Table 2-1. To edit a file, run any of these 
commands: 


> nano myfile 
> emacs myfile 
> vim myfile 


If myfile doesn't exist, the editor creates it. 


Linux also has fine programs to edit Microsoft Office docu- 
ments: LibreOffice (all documents), AbiWord (Word only), and 
Gnumeric (Excel only). Some are probably included in your 
distro, or you can find them easily through web search. 
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Creating a File Quickly 


Text editors create files on request, but you can also create 
an empty file (for later editing) at the command line with the 
touch command: 


> touch newfile Create newfile if it doesn't exist 


or use output redirection (see “Input, Output, and Redirection” 
on page 43) to create a file with or without contents:! 


> echo -n > newfile2 = Echo the empty string to a file 


> > newfile3 Redirect nothing to a file 
> ls > newfile4 Redirect a command's output to a file 
Your Default Editor 


Various Linux programs run an editor when necessary. For 
example, your email program may invoke an editor to compose 
a new message, and less invokes an editor if you press “v.’ 
Usually the default editor is nano or Vim. But what if you want 
a different default editor? Set the environment variables VISUAL 
and EDITOR to your choice, for example: 


> EDITOR=emacs 
> VISUAL=emacs 
> export EDITOR VISUAL 


Both variables are necessary because different programs check 
one variable or the other. Set them in a bash configuration file 
if you want your choices to stick. Any program can be your 
default editor if it accepts a filename as an argument. 


Regardless of how you set these variables, I recommend learn- 
ing basic commands for all three editors in case another pro- 
gram unexpectedly invokes one on a critical file. 


1 The -n option of echo suppresses the newline character. 
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nano -file --opt --help — --version 
nano [options] [files] 


Nano is a basic text editor included in most Linux distros. To 
invoke nano to edit a file, run: 


> nano myfile 
To list all nano keystrokes and commands, press “CG. 


Nano commands generally involve holding down the control 
key and typing a letter, such as “o to save and “x to quit. Nano 
helpfully displays common commands at the bottom of its edit 
window, though some of the vocabulary is a little obscure. (For 
example, nano uses the term “WriteOut” to mean “save file.”) 
Other commands involve the meta key, which is usually the 
Escape key or the Alt key. Nano’s own documentation notates 
the meta key as M- (as in M-F to mean “use the meta key and 
type F”), so this book does the same. For basic keystrokes, see 
Table 2-1. For more documentation, visit https://oreil.ly/p_Pvk. 


emacs -file --opt --help — --version 
emacs [options] [files] 


Emacs is an extremely powerful editing environment with 
thousands of commands, including a rich programming lan- 
guage to define your own editing features. To invoke Emacs to 
edit a file, run: 


> emacs myfile 


On a graphical desktop, Emacs opens a window. To launch 
Emacs without opening a window, run: 


> emacs -nw myfile 


Once Emacs is running, invoke its built-in tutorial by pressing 
“h t. 
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Most Emacs keystroke commands involve the control key (like 
AF) or the meta key, which is usually the Escape key or the Alt 
key. Emacs’s own documentation notates the meta key as M- (as 
in M-F to mean “use the meta key and type F”), so this book 
does the same. For basic keystrokes, see Table 2-1. 


vim -file --opt --help — --version 
vim [options] [files] 


Vim is an enhanced version of the old standard Unix editor vi. 
To invoke Vim to edit a file, run: 


> vim myfile 


Normally, Vim runs in an existing shell or shell window. To 
launch Vim on a graphical desktop in a new window, run: 


> gvim myfile 
To run the Vim tutorial from the shell, run: 
> vimtutor 


Vim operates in two modes, insert and command, and you 
switch between them while editing. Insert mode is for entering 
text in the usual manner, while command mode is for deleting 
text, copy/paste, and other operations. For basic keystrokes in 
command mode, see Table 2-1. 


Table 2-1. Basic keystrokes in text editors 


Task Emacs Nano Vim 
Type text Just type Just type Switch to 
insert mode if 


necessary, by 
typing i, then 
type any text 


Save and quit Axs then Sothen*x — :wq 
ai 
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Task 


Quit without saving 


Save 


Save As 


Undo 

Suspend editor (not in X) 
Switch to insert mode 

Switch to command mode 
Switch to command-line mode 
Abort command in progress 


Move forward 


Move backward 


Move up 


Move down 


Move to next word 

Move to previous word 
Move to beginning of line 
Move to end of line 

Move down one screen 


Move up one screen 


Emacs 


”xAc then 
respond “no” 
when asked 
to save 
buffers 


AXA: 


Ax4w 


A/orx u 
AZ 

(N/A) 

(N/A) 

M-x 

“g 

Af or right 
arrow 


Ab or left 
arrow 


“porup 
arrow 


An or down 
arrow 


M-f 
M-b 
4a 
*e 
Av 


M-v 


Nano 


4x then 
respond “no” 
when asked 
to save 


rao] 


o, then type 
a filename 


M-u 
NZ 
(N/A) 
(N/A) 
(N/A) 
“c 


Af or right 
arrow 


Ab or left 
arrow 


Ap orup 
arrow 


An or down 
arrow 


ASPACE 
M-SPACE 
4a 

we 

Av 


“y 


Vim 


:q! 


2W 


:w filename 


ESC 


Lorright 
arrow 


h or left arrow 


k or up arrow 


3 or down 
arrow 
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Task Emacs Nano Vim 


Move to beginning of M-< M-\ gg 
document 
Move to end of document M-> M-/ G 
Delete next character Ad Ad x 
Delete previous character BACKSPACE BACKSPACE X 
Delete next word M-d (N/A) de 
Delete previous word M- (N/A) db 
BACKSPACE 
Delete current line Aadk 4k dd 
Delete to end of line Ak Aka D 
Define region (type this ASPACE AA (Ctrl Vv 
keystroke to mark the caret) 


beginning of the region, then 
move the cursor to the end of 
the desired region) 


Cut region Aw Ak d 
Copy region M-w -* y 
Paste region ay Au p 
Get help Ah ‘g shelp 
View the manual Ahi ‘g shelp 


* Only if nano’s cut-to-end feature is enabled by pressing M-k. 


Properties of Files 

stat Display attributes of files and directories. 
we Count bytes, words, and lines ina file. 

du Measure disk usage of files and directories. 
file Identify (guess) the type ofa file. 
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mimetype Identify (guess) the MIME type ofa file. 


touch Change timestamps of files and directories. 
chown Change owner of files and directories. 

chgrp Change group ownership of files and directories. 
chmod Change protection mode of files and directories. 
umask Set a default mode for new files and directories. 


lsattr List extended attributes of files and directories. 


chattr Change extended attributes of files and directories. 


When examining a Linux file, keep in mind that the contents 
are only half the story. Every file and directory also has 
attributes that describe its owner, size, access permissions, and 
other information. The ls -1 command (see “Basic File Oper- 
ations” on page 61) displays some of these attributes. Other 
commands let you view and change these and other attributes. 


stat stdout -opt --help — --version 
stat [options] files 


The stat command lists the attributes of files (by default) or 
filesystems (-f option): 


> stat myfile 
File: ‘myfile’ 


Size: 1168 Blocks: 8 
IO Block: 4096 regular file 
Device: 811h/2065d Inode: 37224455 Links: 1 


Access: (0644/-rw-r--r--) Uid: ( 600/lisa) 
Gid: ( 620/users) 

Access: 2015-11-07 11:15:14.766013415 -0500 

Modify: 2015-11-07 11:15:14.722012802 -0500 

Change: 2015-11-07 11:15:14.722012802 -0500 

Birth: - 
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Output includes the filename, size in bytes (1168), size in 
blocks (8), file type (Regular File), permissions in octal (0644), 
permissions in the format of “Is -I” (-rw-r--r--), owner’s user 
ID (600), owner's name (lisa), owner’s group ID (620), owner's 
group name (users), device type (811 in hexadecimal, 2065 in 
decimal), inode number (37224455), number of hard links (1), 
and timestamps of the file’s most recent access,” modification, 
status change (of file permissions or other metadata), and file 
creation (“birth”) if available. The output of stat -f for a 
filesystem is: 


> stat -f myfile 
File: "myfile" 
ID: f02ed2bb86590cc6 Namelen: 255 
Type: ext2/ext3 
Block size: 4096 Fundamental block size: 4096 
Blocks: Total: 185788077 Free: 108262724 
Available: 98819461 
Inodes: Total: 47202304 Free: 46442864 


The output includes the filename (myfile), filesystem ID 
(f02ed2bb86590cc6), maximum allowable length of a filename 
for that filesystem (255 bytes), filesystem type (ext), block size 
for the filesystem (4096), the counts of total, free, and available 
blocks in the filesystem (185788077, 108262724, and 98819461, 
respectively), and the counts of total and free inodes (47202304 
and 46442864, respectively). 


The -t option presents the same data but on a single line, 
without headings. This format is handy for parsing the output: 


> stat -t myfile 
myfile 1168 8 81a4 600 620 811 37224455 10 0 
1446912914 1446912914 1446912914 0 4096 


> size=$(stat -t myfile | cut -d' ' -f2) 
> echo $size The second value 
1168 


2 Most filesystems no longer update access time, or they do so only 
when the file is modified. 


Properties of Files | 85 


Useful options 


-L_ Follow symbolic links and report on the file they point to. 
-f Report on the filesystem containing the file, not the file itself. 


-t  Terse mode: print information on a single line. 


Wc stdin stdout -file --opt --help  --version 
wc [options] [files] 

The we (word count) command prints a count of bytes, words, 
and lines in (presumably) a text file: 


> we myfile 
18 211 1168 myfile 


This file has 18 lines, 211 whitespace-delimited words, and 
1168 bytes. 


Useful options 


-L_ Print the line count only. 
-w Print the word count only. 
-c_ Print the byte count only. 


-L Locate the longest line in each file and print its length in bytes. 


du stdout --opt -help  --version 
du [options] [files | directories] 


The du (disk usage) command measures the disk space occu- 
pied by files or directories. By default, it measures the current 
directory and all its subdirectories, printing totals in blocks for 
each, with a grand total at the end: 
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> du 
36 
340 
40 
416 
216 


./Mail 
./Files/mine 
./Files/bob 
./Files 

. /PC 

2404 . 


Grand total, in blocks 


It can also measure the size of files: 


> du myfile emptyfile hugefile 


4 myfile 

0 emptyfile 

18144 hugefile 

Useful options 

-b Measure usage in bytes. 

-k Measure usage in KB. 

-m Measure usage in MB. 

-BN _ Display sizes in blocks that you define, where 1 block = W bytes. (Default 
= 1024) 

-h Print in human-readable units. For example, if two directories are of size 

-H 1 gigabyte or 25 KB, respectively, du -h prints 1G and 25K. The -h 
option uses powers of 1024, whereas -H uses powers of 1000. 

-L Follow symbolic links and measure the files they point to. 

-C Print a total in the last line. This is the default behavior when measuring 
a directory, but to measure individual files, provide -c if you want a 
total. 

-s Print total sizes only, not the sizes of subdirectories. 
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file stdout -file --opt --help  --version 
file [options] files 


The file command reports the type of a file. The output is an 
educated guess based on the file content and other factors: 


> file /etc/hosts /usr/bin/who letter .docx 
/etc/hosts: ASCII text 

/usr/bin/who: ELF 64-bit LSB executable ... 
letter.docx: Microsoft Word 2007+ 


Useful options 
-b Omit filenames (left column of output). 
-i Print MIME types for the file, such as “text/plain” or “audio/ 


mpeg”, instead of the usual output. 


-f name_file Read filenames, one per line, from the given name_fie, 
and report their types. Afterward, process filenames on the 
command line as usual. 


-L Follow symbolic links, reporting the type of the destination 
file instead of the link. 


-Z Ifa file is compressed (see “Compressing, Packaging, and 
Encrypting” on page 126), examine the uncompressed 
contents to decide the file type, instead of reporting 
“compressed data.” 


mimetype stdout --opt --help  --version 
mimetype [options] files 


The mimetype command, like the file -i command, prints the 
file's MIME type such as text/plain or application/pdf, but 
has more options. 


> mimetype photo.jpg sample.pdf zipfile.zip 
photo.jpg: image/jpeg 
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sample.pdf: application/pdf 
zipfile.zip: application/zip 


Useful options 
-b Omit the leading filenames from the output. 
-d Print longer descriptions like “JPEG image” or “ZIP archive.” 


-l language When provided with the -d option, print the file types in the 
given language. Language codes are the standard two-letter 
ones such as de for German and pt for Portuguese. 


-L For symbolic links, print the type of the linked file instead of 
the link itself (type inode/symlink). 


touch --opt --help — --version 
touch [options] files 


The touch command changes two timestamps associated with a 
file: its modification time (when the file’s data was last changed) 
and its access time (when the file was last read). To set both 
timestamps to the present moment, run: 


> touch myfile 
You can set these timestamps to arbitrary values, for example: 
> touch -d "November 18 1975" myfile 


If a given file doesn’t exist, touch creates an empty file of that 
name (see “Creating a File Quickly” on page 79). 


Useful options 
-a Change the access time only. 
-m Change the modification time only. 
-¢ If the file doesn’t exist, don’t create it (normally, touch 


creates it). 
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-d timestamp Set the file’s timestamp(s). Many formats are acceptable, 
from “12/28/2001 3pm” to “28-May” (the current year is 
assumed, and a time of midnight) to “next tuesday 13:59” to 
“0” (midnight today). Experiment and check your work with 
stat. Full documentation is available from info touch. 


-t timestamp Seta file's timestamp precisely with the format 
[LCC] YY]MMDDhhmm{ . ss]. CCis the two-digit century, 
YY is the two-digit year, MM is the two-digit month, DD is the 
two-digit day, hh is the two-digit hour, mm is the two-digit 
minute, and ss is the two-digit second. For example, -t 
20230812150047 means August 12, 2023 at 15:00:47. 


chown --opt --help — --version 
chown [options] user_spec files 


The chown command changes the owner of files and directories. 
It requires superuser privileges in most cases. To make user 
“smith” the owner of several files and a directory, run: 


> sudo chown smith myfile myfile2 mydir 


The user_spec parameter may be any of these possibilities: 


user_spec Meaning 


smith An existing username 

1001 A numeric user ID 

smith: party An existing username and group name, separated by 
acolon 

1001:234 Numeric user and group IDs, separated by a colon 

:party An existing group name, preceded by a colon 


--reference=file Thesame user and group values as file file 


90 | Chapter 2: File Commands 


Useful options 


--dereference Follow symbolic links and operate on the files they point to. 


-R Recursively change the owner for a whole directory tree. 


chgrp --opt --help — --version 
chgrp [options] group_spec files 


The chgrp (change group) command sets the group ownership 
of files and directories: 


> chgrp users myfile myfile2 mydir 


The group_spec parameter may be any of these possibilities: 


e A group name or numeric group ID 


e --reference=file, to set the same group ownership as 
another given file 


See “Group Management” on page 192 for more information 
on groups. 


Useful options 


--dereference Follow symbolic links and operate on the files they point to. 


-R Recursively change the group for a whole directory tree. 


chmod --opt --help — --version 
chmod [options] permissions files 


The chmod (change mode) command protects files and directo- 
ries from unauthorized users on the same system, by setting 
access permissions. Typical permissions are read, write, and 
execute, and they may be limited to the file owner, the file’s 


Properties of Files | 91 


group owner, and/or all other users. The permissions argument 
can take three different forms: 


e --reference=file, to set the same permissions as another 
given file. 


e An octal (base 8) number, up to four digits long, that 
specifies the file's absolute permissions in bits, as in Fig- 
ure 2-2. The leftmost digit is special (described later) and 
the second, third, and fourth represent the file’s owner, the 
file's group, and all users, respectively. 


e One or more symbolic strings specifying absolute or rela- 
tive permissions (i.e., relative to the file’s current permis- 
sions). For example, a+r makes a file readable by all users. 


The most common octal permissions are: 


v 


chmod 600 myfile Private file for you 


> chmod 644 myfile Everyone can read; you can write 
> chmod 700 mydir Private directory for you 
> chmod 755 mydir Everyone can read; you can write 


Binary: 000 110 100 000 
Symbolic: ss t rwx rwx rwx 


[as 
Special User Group Other 
attributes (u) (9) (0) 
a a 
All 
(a) 


Figure 2-2. File permission bits explained 


In the third, symbolic form, each string consists of three parts: 


Scope (optional) 
u for user, g for group, o for other users not in the group, a 
for all users. The default is a. 
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Command 
+ to add permissions; - to remove permissions; or = to set 
absolute permissions, ignoring existing ones. 


Permissions 
r for read, w for write/modify, x for execute (for directo- 
ries, this is permission to cd into the directory), X for 
conditional execute (explained later), u to duplicate the 
user permissions, g to duplicate the group permissions, o 
to duplicate the “other users” permissions, s for setuid (set 
user ID) or setgid (set group ID), and t for the sticky bit. 


For example, ug+rw would add read and write permission for 
the user and the group, a-x (or just -x) would remove execute 
permission for everyone, and o=r would directly set the “other 
users” permissions to read-only. Combine these strings by sep- 
arating them with commas, such as ug+rw,a-x,0=r. 


Conditional execute permission (X) means the same as x, 
except that it succeeds only if the file is already executable or is 
a directory. Its convenient during large-scale chmod operations 
to change the execute bit on directories but not on files. 


Setuid and setgid, when applied to executable files (programs 
and scripts), have a powerful effect. Suppose you have an exe- 
cutable file F owned by user “smith” and the group “friends.” If 
file F has setuid enabled, then anyone who runs F “becomes” 
user “smith,” with all their rights and privileges, for the dura- 
tion of the program. Likewise, if F has setgid enabled, anyone 
who executes F becomes a member of the “friends” group for 
the duration of the program. As you might imagine, setuid and 
setgid can impact system security, so don't use them unless you 
really know what you're doing. One misplaced chmod +s can 
leave your system vulnerable to attack. 


The sticky bit, most commonly used for /tmp directories, 
controls removal of files in that directory. Normally, if you 
have write permission in a directory, you can delete or move 
files within it, even if you don't have this access to the files 
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themselves. Inside a directory with the sticky bit set, you need 
write permission on a file to delete or move it. 


Useful option 


-R_ Recursively change permissions for a whole directory tree. 


umask stdout -opt --help — --version 
umask [options] [mask] 


The umask command sets or displays your shell’s default mode 
for creating files and directories—whether they are readable, 
writable, and/or executable by yourself, your group, and the 
world. Print your mask: 


> umask Octal output 
0002 
> umask -S Symbolic output 


U=rWX ,g=CWXx,O=rx 


Running umask is a technical operation that requires binary and 
octal math, but let’s begin with simple recipes. Use mask 0022 
to give yourself full privileges on new files and directories, and 
give all other users read and execute privileges only: 


> umask 0022 

> touch newfile && mkdir newdir 

> ls -1dG newfile newdir 

-rw-r--r-- 1 smith 0 Nov 11 12:25 newfile 
drwxr-xr-x 2 smith 4096 Nov 11 12:25 newdir 


Use mask 0002 to give yourself and your default group full 
privileges, and read/execute to others: 


> umask 0002 

> touch newfile2 && mkdir newdir2 

> 1s -1dG newfile2 newdir2 

-rw-rw-r-- 1 smith 0 Nov 11 12:26 newfile2 
drwxrwxr-x 2 smith 4096 Nov 11 12:26 newdir2 
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Use mask 0077 to give yourself full privileges with nothing for 
anyone else: 


> umask 0077 

> touch newfile3 && mkdir newdir3 

> ls -1dG newfile3 newdir3 

-[W------- 1 smith © Nov 11 12:27 newfile3 
drwx------ 2 smith 4096 Nov 11 12:27 newdir3 


Now, the technical explanation. A umask is a binary (base two) 
value, though it is commonly presented in octal (base eight). 
It defines your default protection mode by combining with the 
octal value 0666 for files and 0777 for directories, using the 
binary operation NOT AND. For example, the umask 0002 yields 
a default file mode of 0664: 


0666 NOT AND 0002 

= 000110110110 NOT AND 000000000010 
= 000110110110 AND 111111111101 
000110110100 

= 0664 


Similarly for directories, 0002 NOT AND 0777 yields a default 
mode of 0775. 


Isattr stdout --opt --help 
lsattr [options] [files | directories] 


Linux files have additional attributes beyond their access per- 
missions. For files on an “ext” filesystem (ext3, ext4, etc.), list 
these extended attributes with the lsattr (list attribute) com- 
mand and change them with chattr. For example, this file is 
immutable (i) and undeletable (u): 


> lsattr attrfile 
-u--i--- attrfile 


With no files specified, lsattr prints the attributes of all files in 
the current directory. Attributes include: 
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Attribute Meaning 


a Append-only: appends are permitted to this file, but it cannot 
otherwise be edited. Root only. 


A Accesses not timestamped: accesses to this file don’t update its access 
timestamp (atime). 

c Compressed: data is transparently compressed on writes and 
uncompressed on reads. 

d Don’t dump: tell the dump program to ignore this file when making 
backups (see “Backups and Remote Storage” on page 241). 

sf Immutable: file cannot be changed or deleted (root only). 

j Journaled data (on filesystems that support journaling). 

s Secure deletion: if deleted, this file’s data is overwritten with zeros. 

S Synchronous update: changes are written to disk immediately. 

u Undeletable: file cannot be deleted. 


Before using this command seriously, read the manpage for 
details, especially for filesystem-specific issues. 


Useful options 


-R_ Recursively process directories. 
-a__Listall files, including those whose names begin with a dot. 


-d_ Iflisting a directory, do not list its contents, just the directory itself. 


chattr --opt --help 
chattr [options] [+|-|=]attributes [files] 


The chattr command changes a file’s extended attributes, the 
same ones that lsattr displays. With syntax similar to the 
chmod command, chattr adds (+) or removes (-) attributes 
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relatively, or sets attributes absolutely (=). For example, to pre- 
vent a file from deletion even by the superuser, run: 


> sudo chattr +i attrfile Set attribute i 

> sudo rm attrfile Deletion fails 

rm: cannot remove ‘attrfile': Operation not permitted 
> sudo chattr -i attrfile Unset attribute i 

> sudo rm attrfile Deletion succeeds 


Not all attributes are supported by all filesystems. Read the 
manpage for details, especially for filesystem-specific issues. 


Useful option 


-R_ Recursively process directories. 


Locating Files 


find Locate files in a directory hierarchy. 

xargs Turn a list of files into a list of commands (and much more). 
locate Create an index of files, and search the index for a string. 
which Locate executables in your search path (command). 

type Locate executables in your search path (bash built-in). 


whereis Locate executables, documentation, and source files. 


A typical Linux system has hundreds of thousands of files, so 
Linux includes commands for locating files of interest. find is a 
brute-force command that slogs file by file through a directory 
tree to locate a target. Locate is much faster, searching through 
a prebuilt index that you generate as needed. (Some distros 
generate the index nightly by default.) 


To locate programs in the filesystem, the which and type 
commands check all directories in your shell search path. 
type is built into the bash shell, while which is a program 
(normally /usr/bin/which); type is faster and can detect shell 
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aliases.? In contrast, whereis examines a known set of directo- 
ries, rather than your search path. 


find stdout --help — --version 
find [directories] [expression] 


The find command searches one or more directories (and their 
subdirectories recursively) for files matching certain criteria. 
It is very powerful, with over 50 options, and unfortunately, 
a rather unusual syntax. Here are some simple examples that 
search the entire filesystem from the current directory (indica- 
ted by a dot): 


Find a particular file named myfile: 
> find . -type f -name myfile -print 
./myfile 


Print filenames beginning with “myfile”. Notice that the asterisk 
is escaped so the shell ignores it for pattern-matching and it’s 
passed literally to the find command: 


> find . -type f -name myfile\* -print 
./myfile3 
./myfile 
./myfile2 


Print all directory names: 


> find . -type d -print 
./jpegexample 
./dir2 


./mydir 
./mydir/dir 


3 The tcsh shell performs some trickery to make which detect aliases. 
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Useful options 


-name pattern 
-path pattern 
-lname pattern 


-iname pattern 
-ipath pattern 
-ilname pattern 


-regex regexp 


-type t 


-atime N 
-ctime N 
-mtime N 


-amin N 
-cmin N 
-mmin N 


-anewer other_file 
-cnewer other_file 
-newer other_file 


-maxdepth NV 
-mindepth NV 


- follow 


Given a file pattern, find files with names (- name), 
pathnames (- path), or symbolic link targets 

( -Lname) that match the pattern, which may 
include shell pattern-matching characters *, ?, and 
[]. (You must escape the patterns, however, so 
they are ignored by the shell and passed literally to 
find.) Paths are relative to the directory tree being 
searched. 


These options are the same as - name, - path, and 
- Lname, respectively, but are case-insensitive. 


The path (relative to the directory tree being 
searched) must match the given regular expression. 


Locate only files of type f. Types include plain files 
(f), directories (d), symbolic links (1), block devices 
(b), character devices (c), named pipes (p), and 
sockets (s). 


File was last accessed (- atime), last modified 
(-mtime), or hada status change (-ctime) 
exactly N *24 hours ago. Use +N for “greater than 
N,” or -N for “less than W.” 


File was last accessed (- amin), last modified 
(-mmin), or had a status change (- cmin) exactly WV 
minutes ago. Use +N for “greater than N,,"or -N for 
“less than WN." 


File was accessed (- anewer), modified (- newer), 
or had a status change (- cnewer) more recently 
than other_file. 


Consider files at least (-mindepth) or at most 
(-maxdepth) W levels deep in the directory tree 
being searched. 


Dereference symbolic links. 
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-depth Proceed using depth-first search: completely search 
a directory’s contents (recursively) before operating 
on the directory itself. 


-xdev Limit the search to a single filesystem (i.e., don’t 
cross device boundaries). 
-size N [bckw] Consider files of size N, which can be given in blocks 


(b), one-byte characters (c), KB (k), or two-byte 
words (w). Use +N for “greater than WN,” or -N’ for 


“less than N.” 
-empty File has zero size and is a regular file or directory. 
-user name File is owned by the given user. 
-group name File is owned by the given group. 
-perm mode File has permissions equal to mode. Use - mode to 


check that al! of the given bits are set, or +mode to 
check that any of the given bits are set. 


These operators group or negate parts of the expression: 


expression1 -a expression2 
And. (This is the default if two expressions appear side by 
side, so the -a is optional.) 


expression1 -o expression2 
Or. 


! expression 
-not expression 
Negate the expression. 


( expression ) 
Precedence markers, just like in algebra class. Evaluate 
what’s in parentheses first. You may need to escape these 
from the shell with “\” 


expressionl , expression2 
Comma operator. Evaluate both expressions and return 
the value of the second one. 
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Once you've specified the search criteria, tell find to perform 
the following actions on files that match the criteria: 


-print 


-printf string 


-printd 


-exec cmd ; 


-ok cmd ; 


-ls 


-delete 


xargs 


Print the path to the file, relative to the search directory. 


Print the given string, which may have substitutions applied to 
itin the manner of the C library function, printf (). See the 
manpage for the full list of outputs. 


Like - print, but instead of separating each line of output 
with a newline character, use a null (ASCII 0) character. 
Use when piping the output of find to another command 
and your list of filenames may contain space characters. Of 
course, the receiving command must be capable of reading 
and parsing these null-separated lines (e.g.,xargs -0). 


Invoke the given shell command, cmd. Escape any shell 
metacharacters, including the required, final semicolon, so 
they aren't evaluated by the shell. Also, the symbol {} 
(quoted or escaped) represents the path to the file found. 
A full example is: 

find . -exec ls '{}' \; 


Same as - exec, but also prompts the user before each 
invocation of the command cmd. 
Perform the command 1s -dils on each file. 


Perform the command rm on each file. (Careful!!) 


stdin stdout --help —--version 


xargs [options] [command] 


xargs is one of the oddest yet most powerful commands avail- 
able to the shell. It reads lines of text from standard input, 
turns them into commands, and executes them. This might not 
sound exciting, but xargs has some unique uses, particularly 
for processing a list of files you've located. Suppose you made a 
file named important that lists important files, one per line: 
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> cat important 
/home/jsmith/mail/love-letters 
/usr/local/lib/critical_stuff 
/etc/passwordfile2 


xargs can process each of these files easily with other Linux 
commands. For instance, the following command runs the ls 
-L command on all the listed files: 


> cat important | xargs ls -l 

Similarly, view the files with less: 

> cat important | xargs less 

and even delete them with rm: 

> cat important | xargs rm -f = Careful! Destroys files!! 


In each case, xargs reads the file important and produces 
and runs new Linux commands. The power begins when the 
input list doesn’t come from a file, but from another command 
writing to standard output. In particular, the find command, 
which prints a list of files, makes a great partner for xargs. For 
example, to search your current directory hierarchy for files 
containing the word “tomato”: 


> find . -type f -print | xargs grep -l tomato 
./findfilet 

./findfile2 

> cat findfilel findfile2 

This file contains the word tomato. From findfile1 
Another file containing the word tomato. From findfile2 


This command has a problem, however: it mishandles file- 
names that contain whitespace, like my stuff. If find prints this 
filename, then xargs constructs an incorrect grep command: 


grep -l tomato my stuff 


which tells grep to process two files named my and stuff. Oops! 
Now imagine that the command you passed to xargs was rm 
instead of grep. Then rm would delete the wrong files! To avoid 
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this problem, tell find and xargs to use a non-whitespace char- 
acter between lines of text—specifically, a null character (ASCII 
value zero). The command find -print0 ends lines with nulls, 
and xargs -0 expects nulls. 


> find . -type f -printO | xargs -0 grep -l tomato 


I’ve barely scratched the surface of the xargs command, so I 
hope you'll continue experimenting. (For safety, tell xargs to 
invoke harmless commands like grep and ls for your tests, 
rather than destructive commands like rm.) 


Useful options 


-n_ k_ Feed kK lines of input to each executed command. The common -n1 
guarantees that each execution processes only one line of input. 
Otherwise, xargs may pass multiple lines of input to a single command. 


-0 Set the end-of-line character for input to be ASCII zero rather than 
whitespace, and treat all characters literally. Use this when the input 
comes from find -printo. 


xargs versus Command Substitution 


If you remember “Quoting” on page 46, you might realize 
that some xargs tricks can be accomplished with command 
substitution: 


> cat file_list | xargs ls -l = With xargs 
>ls -l $(cat file_list) With $() 
> ls -l ‘cat file_list” With backquotes 


While these commands do similar things, the last two can fail if 
the output of cat becomes longer than the maximum length of 
a shell command line. The xargs solution reads unlimited text 
from standard input rather than the command line, so it’s more 
scalable for large or risky operations. 
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locate stdout -opt --help — --version 
locate [options] 


The locate command, with its partner updatedb, creates an 
index (database) of file locations that is quickly searchable.* 
If you plan to search for many files over time in a directory 
hierarchy that doesn't change much, locate is a good choice. 
To locate a single file or perform more complex processing of 
found files, use find. 


Some distros automatically index the entire filesystem regularly 
(e.g., once a day), so you can simply run locate and it works. 
But if you ever need to create an index yourself of a directory 
and all its subdirectories (say, ~/linuxpocketguide), storing the 
index in /tmp/myindex, run: 


> updatedb -10 -U ~/linuxpocketguide -o /tmp/myindex 


(Note that -10 is a lowercase L followed by a zero, not the 
number 10.) Then search for a string in the index, such as 
“myfile”: 


> locate -d /tmp/myindex myfile 

/home/dbarrett/linuxpocketguide/myfile 
/home/dbarrett/linuxpocketguide/myfile2 
/home/dbarrett/linuxpocketguide/myfile3 


updatedb has an interesting, optional security feature. If you 
run it as root, you can create an index that displays only files 
that the user is permitted to see, based on file permissions. 
Simply add sudo and omit the -10 option: 


> sudo updatedb -U directory -o /tmp/myindex 


4 My locate command comes from a package called “plocate.” Some 
systems have older packages called “mlocate” or “slocate” with slightly 
different usage. If you have slocate, simply run slocate instead of 
updatedb in my examples. 
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Indexing options for updatedb 


-u Create index from the root directory downward. 
-Udirectory Create index from directory downward. 

-L (0[1) Turn security off (0) or on (1). The default is 1. 

-e paths Exclude paths from the index, separated by whitespace. 


-ooutfile Write the index to file outfile. 


Search options for locate 


-d index Indicate which index to use (in my example, /mp/myindex). 
-i Case-insensitive search. 


-r regexp Search for filenames matching the given regular expression. 


which stdout --opt --help  --version 
which file 


The which command locates an executable file in your shell’s 
search path. To locate the who command, run: 


> which who 
/usr/bin/who 


You can even find the which program itself: 


> which which 
/usr/bin/which 


If several programs in your search path have the same name 
(e.g., /bin/who and /usr/bin/who), which reports only the first. 
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type stdout -- opt --version 
type [options] commands 


The type command, like which, locates an executable file in 
your shell’s search path: 


> type grep who 
grep is /bin/grep 
who is /usr/bin/who 


type also identifies shell features, such as aliases and built-in 
commands: 


> type which type rm if 
which is /usr/bin/which 

type is a shell builtin 

rm is aliased to ‘/bin/rm -i' 
if is a shell keyword 


As a built-in command, type is faster than which, but it’s avail- 
able only in certain shells such as bash. 


whereis stdout 
whereis [options] files 


The whereis command attempts to locate the given files by 
searching a hardcoded list of directories. It can find executa- 
bles, documentation, and source code. whereis is somewhat 
quirky because its internal list of directories might not include 
the ones you need. 


> whereis nano 
nano: /usr/bin/nano /usr/share/nano ... 
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Useful options 


-b 
-7 
=S 
-Bdirs... -f 
-Mdirs... - 
-Sdirs... -f 


List only binary executables (- b), manpages (-m), or source code 
files (-s). 


Search for binary executables (-B), manpages (-M), or source 
code files (- S) in the given directories. You must follow the 
directory list with the - f option before listing the files you seek. 


Manipulating Text in Files 


grep 
cut 
paste 
column 
tr 
expand 
unexpand 
sort 
uniq 

tac 

shuf 


tee 


Find lines in a file that match a regular expression. 
Extract columns from a file. 

Append text from multiple files in columns. 
Organize text into columns. 

Translate characters into other characters. 
Convert from tabs to spaces. 

Convert from spaces to tabs. 

Sort lines of text by various criteria. 

Locate identical lines in a file. 

Reverse a file line by line. 

Randomly shuffle the lines of a file (permutation). 


Write to a file and print on standard output, simultaneously. 


Perhaps Linux’s greatest strength is text manipulation: massag- 
ing a text file (or standard input) into a desired form by apply- 
ing transformations, often in a pipeline. Many commands do 
this, but here I focus on some of the most important tools for 
transforming text. 
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grep stdin stdout -file --opt --help  --version 
grep [options] pattern [files] 


The grep command is one of the most consistently useful and 
powerful in the Linux arsenal. Its premise is simple: given one 
or more files, print all of their lines that match a regular expres- 
sion. For example, if a file randomlines contains these lines: 


The quick brown fox jumped over the lazy dogs! 
My very eager mother just served us nine pancakes. 
Film at eleven. 


and you print all lines containing “pancake’, you get: 


> grep pancake randomlines 
My very eager mother just served us nine pancakes. 


Now use a regular expression to print lines ending in an excla- 
mation point: 


> grep '!$' randomlines 
The quick brown fox jumped over the lazy dogs! 


grep can use two different types of regular expressions: basic 
and extended. They are equally powerful, just different, and 
their syntax is in Table 2-2. Regular expressions are well worth 
your time to learn. Other powerful Linux programs use them 
too, such as sed and perl. 


Useful options 


-V Print only lines that do not match the regular expression. 

-1 Print only the names of files that contain matching lines, not the 
lines themselves. 

-L Print only the names of files that do not contain matching lines. 

-c Print only a count of matching lines. 

-0 Print only the strings that match, not whole lines. 

-n In front of each line of matching output, print its original line 


number. (Or print the byte offset with - b.) 
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-i Case-insensitive match. 


-w Match only complete words. 

-xX Match only complete lines. Overrides -w. 

-a Treat all files as plain text. Sometimes grep mistakenly treats a file 
as binary and won't print matching lines. 

-A N After each matching line, print the next A lines from its file. 

-B N Before each matching line, print the previous NV lines from its file. 


-C N Same as -A N -B W: print N lines (from the original file) above and 
below each matching line. 


--color Highlight the matched text in color, for better readability. 


-r Recursively search all files in a directory and its subdirectories. 

-R Same as -r but also follow all symbolic links. 

-E Use extended regular expressions. See egrep. 

-F Use lists of fixed strings instead of regular expressions. See fgrep. 


Table 2-2. Regular expressions for grep (plain) and egrep (extended) 
Plain Extended Meaning 


Any single character. 


[vend [se] Any single character in this list. 

[4...] [ic] Any single character not in this 
list. 

(28) (aa) Grouping. 

\| | Or. 

A a Beginning of a line. 

$ $ End of a line. 

\< \< Beginning of a word. 

\> \> End of a word. 

[:alnum: ] [:alnum:] — Anyalphanumeric character. 
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supper: 


\+ 


\? 


\{n\} 


\{n\} 


\{n,m\} 


\c 


] 
7] 
] 


:xdigit: ] 


Extended 


[: 
[: 
[: 
[: 
[: 
[: 
[ie 
[: 
[iz 
lies 


* 


alpha: ] 
cntrl:] 
digit: 
graph: | 
lower: 
print: ] 
punct: 
space: 
upper: 
xdigit: ] 


{n,m} 


\c 


] 
] 
] 


Meaning 

Any alphabetic character. 
Any control character. 
Any digit. 

Any graphic character. 
Any lowercase letter. 
Any printable character. 
Any punctuation mark. 
Any whitespace character. 
Any uppercase letter. 
Any hexadecimal digit. 


Zero or more repetitions of a 
regular expression. 


One or more repetitions of a 
regular expression. 


Zero or one occurrence of a 
regular expression. 


Exactly n repetitions of a regular 
expression. 


nor more repetitions of a regular 
expression. 


Between n and m (inclusive) 
repetitions of a regular 
expression, n<m. 


The literal character c. For 
example, use \* to match 

an asterisk or \\ to match 

a backslash. Alternatively, use 
square brackets, like [*] or [\]. 
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egrep stdin stdout -file --opt --help  --version 
egrep [options] pattern [files] 


The egrep command runs grep with the option -E£ pre-applied 
to use extended regular expressions.° 


fgrep stdin stdout -file --opt --help  --version 
fgrep [options] [fixed_strings] [files] 


The fgrep command runs grep with the option -F pre-applied.° 
Instead of regular expressions, fgrep accepts a list of fixed 
strings separated by newlines. For example, if you have a dictio- 
nary file full of strings, one per line: 


> cat my_dictionary_file 
aardvark 

abbey 

abbot 


fgrep searches for those strings in one or more input files: 


> fgrep -f my_dictionary_file story 
a little aardvark who went to 
visit the abbot at the abbey. 


fgrep is convenient when searching for nonalphanumeric char- 
acters like * and { because they are treated literally as fixed 
strings, not as regular expression characters. 


It’s easiest for fgrep to read the fixed strings from a file, but it 
can also take them from the command line if you quote them. 
To search for the strings “one’, “two”, and “three” in a file, run: 


5 The commands egrep and fgrep are officially deprecated, but they are 
so deeply embedded in a million Linux shell scripts that I don’t think 
they'll ever go away. 
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> fgrep 'one Note that I am typing newline characters 
two 
three' myfile 


grep and End-of-Line Characters 


When you match the end of a line ($) with grep, text files 
created on Microsoft Windows or macOS systems may produce 
odd results. Each OS has a different standard for ending a line. 
On Linux, each line in a text file ends with a newline character 
(ASCII 10). On Windows, text lines end with a carriage return 
(ASCII 13) followed by a newline character. And on macOS, 
a text file might end its lines with newlines alone or carriage 
returns alone. If grep doesn't match the ends of lines properly, 
check for non-Linux end-of-line characters with cat -v, which 
displays carriage returns as “M: 


> cat -v dosfile.txt 
Uh-oh! This file seems to end its lines with*M 
carriage returns before the newlines. 


To remove the carriage returns, use the tr -d command: 


>tr -d '\r' < dosfile.txt > /tmp/linuxfile.txt 
> cat -v /tmp/linuxfile. txt 

Uh-oh! This file seems to end its lines with 
carriage returns before the newlines. 


cut stdin stdout -file --opt --help  --version 
cut -(b|c|f)range [options] [files] 


The cut command extracts columns of text from files. A “col- 
umn” is defined by character offsets (e.g., the 19th character of 
each line): 


> cut -c19 myfile 
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or by byte offsets (which are different from characters if your 
language has multibyte characters): 


> cut -b19 myfile 


or by delimited fields (e.g., the fifth field in each line of a 
comma-delimited file, data.csv): 


> cat data.csv 
one, two, three, four, five,six,seven 
ONE, TWO, THREE, FOUR, FIVE, SIX, SEVEN 


1,25354,556,7 

> cut -f5 -d, data.csv 
five 

FIVE 

5 


You aren't limited to printing a single column: provide a range 
(3-16), a comma-separated sequence (3,4,5,6,8,16), or both 
(3,4,8-16). For ranges, if you omit the first number (-16), a 1 
is assumed (1-16); if you omit the last number (5-), the final 
column is assumed. 


Useful options 
-dc Use character Cas the input delimiter character 
between fields for the - f option. By default it’s a 
tab character. 


--output-delimiter=5 Use string Sas the output delimiter character 
between fields for - f. By default it’s a tab 
character. 


-s Suppress lines that don’t contain any delimiter 
characters at all. Otherwise they're printed 
unchanged. 
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stdin stdout -file --opt --help  --version 
paste 
paste [options] [files] 


The paste command is the opposite of cut: it treats several files 
as vertical columns and combines them on standard output: 


cat letters 


cat numbers 


A 
B 
C 


paste letters numbers 


= 
A 

B 

C 

% 

1 

Z 

3 

4 

5 

> paste numbers letters 
1 

Z 

3 

4 

5 

5 

A 1 

B 

C 


up wWDN 


Useful options 


-d delimiters Print the given delimiter characters between columns; the 
default is a tab character. Provide a single character (-d:) 
to be used always, ora list of characters (- dxyz) to be 
applied in sequence on each line (the first delimiter is x, 
then y, then z, then x, theny, ...). 
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-S Sideways: transpose the rows and columns of output: 
> paste -s letters numbers 
A B C 
1 2 3 4 5 


column stdin stdout -opt --help — --version 
column [options] [files] 


The column command reads lines of text and prints them in 
columns. By default, it creates as many columns as will fit in 
the width of your terminal, so the output may vary. 


> seq 1 18 Print 18 numbers 

1 

2 

3 

18 

> seq 1 18 | column Create columns vertically 


1 4 7 10 13 16 

2 5 8 11 14 17 

3 6 9 12 15 18 

> seq 1 18 | column -x Create columns horizontally 
1 2 3 4 ~=«5 6 

7 8 9 10 11 12 

13 14 15 16 17 18 


column can also convert existing columns into a nicely format- 
ted table: 


> cat threes Original file 

one two three Ragged columns 

Do Re Mi 

you and me 

> column -t threes Format as a table with -t 
one two three Neat columns 

Do Re Mi 


you and me 
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> column -t -N A,B,C threes Add headings with -N 
A B C 

one two three 

Do Re Mi 

you and me 


Useful options 


-sC Use character Cas the input separator between columns. 
-0 C Use character Cas the output separator between columns. 
-t Format the output as a table. 


-N headings Add headings to a table (requires - t). Provide a comma- 
separated list. 


--json Print the output in JSON format (requires - t and -N). 


tr stdin stdout help —--version 


tr [options] charset1 [charset2] 


The tr command performs some simple translations of one set 
of characters into another. For example, to capitalize everything 
ina file: 


> cat wonderfulfile 

This is a very wonderful file. 

> cat wonderfulfile | tr a-z A-Z 
THIS IS A VERY WONDERFUL FILE. 


or to change all vowels into asterisks: 


> cat wonderfulfile | tr aeiouAEIOU '*' 
Th*s *s * v*¥ry w*nd*rf*l f*l*. 


or to delete all vowels: 


> cat wonderfulfile | tr -d aeiouAEIOU 
Ths s_ vry wndrfl fl. 
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Delete all carriage returns from a DOS text file so it’s more 
compatible with Linux text utilities like grep: 


> tr -d '\r' < dosfile.txt > linuxfile.txt 


tr translates the first character in charset] into the first charac- 
ter in charset2, the second into the second, the third into the 
third, and so on. If the length of charset1 is N, only the first 
N characters in charset2 are used. (If charset1 is longer than 
charset2, see the -t option.) 


Character sets can have the following forms: 


ABDG The sequence of characters A, B, D, G. 

A-Z The range of characters from A to Z. 

[x*y] y tepetitions of the character x. 

[ :dass: ] The character classes accepted by grep, suchas [:alnum: ] 


and [: digit: ]. See Table 2-2. 


tr also understands the escape characters “\a” (*G = alert by 
ringing bell), “\b” (4H = backspace), “\f” (*L = formfeed), “\n” 
(43 = newline), “\r” (4M = return), “\t? (41 = tab), and “\v” (4K 
= vertical tab) accepted by printf (see “Screen Output” on page 
275), as well as the notation \nnn to mean the character with 
octal value nnn. 


tr is great for quick and simple translations, but for more 
powerful jobs consider sed, awk, or a programming language 
such as perl. 


Useful options 


-d_ Delete the characters in charset7 from the input. 


-s Eliminate adjacent duplicates (found in charset?) from the input. For 
example, tr -s aeiouAEIOU would squeeze adjacent, duplicate vowels 
to be single vowels (reeeeeeally would become really). 


-c Complement: operate on all characters not found in charset7. 
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-t_ If charset7 is longer than charset2, make them the same length by 
truncating charset7. If - t is not present, the last character of charset2 is 
(invisibly) repeated until charset2 is the same length as charset7. 


expand stdin stdout -file --opt --help  --version 


expand [options] [files] 


unexpand [options] [files] 


The expand command converts tab characters to an equivalent- 
looking number of space characters, and unexpand does the 
opposite. By default, a tab stop occurs every eight spaces, but 
you can change this with options. 


> expand tabfile > spacefile 
> unexpand spacefile > tabfile 


To check whether a file contains spaces or tabs, use the cat -T 
command, which displays tabs as “I, or the od -c command, 
which displays tabs as \t. 


Useful option 


-t N Specify that one tab stop occurs every N spaces. 


sort stdin stdout -file --opt --help  --version 
sort [options] [files] 


The sort command prints lines of text in alphabetical order, 
or sorted by some other rule you specify. All provided files are 
concatenated, and the result is sorted and printed: 


> cat threeletters 
def 
xyz 
abc 
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> sort threeletters 


abc 

def 

XYZ 

Useful options 

-f Case-insensitive sorting. 

-n Sort numerically (i.e., 9 comes before 10) instead of alphabetically (10 
comes before 9 because it begins with a “1” character). 

-g Another numerical sorting method with a different algorithm that, 
among other things, recognizes scientific notation (7.4e3 means “7.4 
times ten to the third power,” or 7400). Run info sort for full 
technical details. 

-u Unique sort: discard duplicate lines. (If used with -c to check sorted 
files, fail if any consecutive lines are identical.) 

-c Don’t sort, just check if the input is already sorted. If itis, print 
nothing; otherwise, print an error message. 

-b Ignore leading whitespace in lines. 

-r Reverse the output: sort from greatest to least. 


-k key Choose sorting keys, described next. (Combine with - t to choose a 
separator character between keys.) 


-t X — UseXas the field separator for the -k option. 


A sorting key indicates a portion of a line to consider when 
sorting, instead of the entire line. Consider this file of names 
and addresses: 


> cat people 

George Washington,123 Main Street,New York 
Abraham Lincoln,54 First Avenue,San Francisco 
John Adams,39 Tremont Street,Boston 


An ordinary sort displays the “Abraham Lincoln” line first. But 
you can sort on the second value (-k2), the address, if you 
consider each line as three comma-separated values (-t,). The 
first address alphabetically is “123 Main Street”: 
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> sort -k2 -t, people 

George Washington,123 Main Street,New York 
John Adams ,39 Tremont Street,Boston 

Abraham Lincoln,54 First Avenue,San Francisco 


Likewise, sort on the third value (-k3), the city, where “Boston” 
is first alphabetically: 


> sort -k3 -t, people 

John Adams ,39 Tremont Street,Boston 

George Washington,123 Main Street,New York 
Abraham Lincoln,54 First Avenue,San Francisco 


The general syntax is -k F1[.C1][,F2[.C2]] which means: 


Item Meaning Default value 
FI Starting field Required 
4] Starting position within field 1 1 
F2 Ending field Last field 
Q Starting position within ending 1 
field 


So sort -k1.5 sorts by the first field beginning at its fifth char- 
acter; and sort -k2.8,5 means “from the eighth character of 
the second field, up to the first character of the fifth field” The 
-t option changes the behavior of -k so it considers delimiter 
characters such as commas rather than spaces. 


Repeat the -k option to define multiple keys. sort applies them 
from first to last as found on the command line. 


uniq stdin stdout -file --opt --help  --version 
uniq [options] [files] 


The uniq command operates on consecutive, duplicate lines of 
text. For example, if you have a file Jetters2: 
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> cat letters2 
a 
b 
b 
c 


b 


then uniq would detect and process (in whatever way you spec- 
ify) the two consecutive bs, but not the third b. By default, uniq 
deletes the consecutive duplicates: 


uniq letters2 


~ 
a 
b Deleted one “b” character 
ce 
b 


unig is often used after sorting a file: 


> sort letters2 | uniq 
a 
b 


Cc 


In this case, only a single b remains because sort placed all 
three adjacently, and then unig collapsed them to one. To count 
duplicate lines instead of eliminating them, use the -c option: 


> sort letters2 | uniq -c 


BPwWPR 
oom) 


Useful options 


-C Count adjacent duplicate lines. 
-i Case-insensitive operation. 

-u Print unique lines only. 

-d Print duplicate lines only. 


-s N_ Skip the first N characters on each line when detecting duplicates. 
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-f N_ Ignore the first V whitespace-separated fields on each line when 
detecting duplicates. 


-W N_ Consider only the first W characters on each line when detecting 
duplicates. If used with - s or - f, uniq ignores the specified number of 
characters or fields first, then considers the next NV characters. 


tac stdin stdout -file --opt --help  --version 
tac [options] [files] 


The tac command, which is “cat” spelled backwards, prints the 
lines of a file in reverse order. 


> cat lines 
one 

two 

three 

> tac lines 
three 

two 

one 


It’s great for reversing lines that are in chronological order, such 
as the contents of a logfile for a Linux service. 


Given multiple filenames as arguments, tac reverses each file in 
turn. To reverse all lines in all files together, combine them first 
with cat and pipe the output to tac: 


> cat myfile myfile2 myfile3 | tac 


shuf stdin stdout --opt --help — --version 
shuf [options] [files] 


The shuf command shuffles (permutes) lines of text randomly, 
from a file or other sources. 
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> cat lines Original file 
one 

two 

three 

> shuf Lines Run it once 
two 

three 

one 

> shuf lines Run it again with different output 
one 

three 

two 


Or, provide strings on the command line with shuf -e: 


> shuf -e apple banana guava 
guava 
apple 
banana 


Or, randomize the numbers in a range with shuf -i: 


> shuf -i 0-3 
3 
1 
0 
2 


The shuf command is great for extracting random subsets of 
lines from a file. For example, given a file of people's first 
names, shuf can randomize the file and can print a set number 
of lines with the -n option: 


> cat names 
Aaron 

Ahmed 

Ali 

Ana 


> shuf -n3 names 
Ying 

Robert 

Patricia 


Manipulating Text in Files | 123 


> shuf -n1 names 
Fatima 


Useful options 


-e Shuffle strings provided on the command line. 
-irange Shuffle a range of whole numbers, such as 1- 10. 
-n K Print at most K lines. 


-r Repeatedly shuffle and print, producing unlimited output. Combine 
with -n to limit the output. 


tee stdin stdout -file --opt --help  --version 
tee [options] files 


Like the cat command, the tee command copies standard 
input to standard output unaltered. Simultaneously, it copies 
the input to one or more files. tee is most often found in the 
middle of pipelines, writing some intermediate data to a file 
while passing it to the next command in the pipeline: 


> who | tee original_who | sort 
barrett pts/1 Sep 22 21:15 
byrnes pts/0 Sep 15 13:51 
silver :0 Sep 23 20:44 
silver pts/2 Sep 22 21:18 


The preceding command line writes the unsorted output of who 
to the file original_who and also passes the same text to sort, 
which produces sorted output on screen: 


> cat original_who 

silver :0 Sep 23 20:44 
byrnes pts/0 Sep 15 13:51 
barrett pts/1 Sep 22 21:15 
silver pts/2 Sep 22 21:18 
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Useful options 


-a_ Append instead of overwriting files. 


-i Ignore interrupt signals. 


More powerful manipulations 


Linux has hundreds of other filters that produce ever more 
complex manipulations of the data. But with great power 
comes a great learning curve, too much for a short book. Here 
are a few of the most capable filters to get you started. 


awk 


AWK is a pattern-matching language. It matches data by regu- 
lar expression and then performs actions based on the data. 
Here are a few simple examples to process a text file, myfile. 


Print the second and fourth word on each line: 
> awk '{print $2, $4}' myfile 
Print all lines that are shorter than 60 characters: 


> awk "Length < 60 {print}' myfile 


sed 


Like AWK, sed is a pattern-matching engine that manipulates 
lines of text. Its syntax is shared by several other Linux pro- 
grams, such as Vim. For example, print a file with all occur- 
rences of the string “me” changed to “YOU”: 


> sed 's/me/YOU/g' myfile 
Print lines 3 through 5, inclusive, from a file: 


> sed -n '3,5p' myfile 


m4 


m4 is a macro processor. It locates keywords within a file and 
substitutes values for them. For example, given this file: 
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> cat substitutions 
My name is NAME and I am AGE years old. 
ifelse(QUOTE,yes,Learn Linux today!) 


see m4 perform substitutions (-D) for NAME, AGE, and QUOTE: 


> m4 -D NAME=Sandy substitutions 
My name is Sandy and I am AGE years old. 


> m4 -D NAME=Sandy -D AGE=25 substitutions 
My name is Sandy and I am 25 years old. 


> m4 -D NAME=Sandy -D AGE=25 -D QUOTE=yes substitutions 
My name is Sandy and I am 25 years old. 
Learn Linux today! 


Perl, PHP, Python, Ruby 


If you need even more powerful text processing, Linux includes 
interpreters for Perl, PHP, Python, Ruby, and other full-fledged 
programming languages. See “Beyond Shell Scripting” on page 
327 for references. 


Compressing, Packaging, and Encrypting 


tar Package multiple files into a single file. 
gzip Compress files with GNU Zip. 

gunzip Uncompress GNU Zip files. 

bzip2 Compress files in BZip format. 

bunzip2 Uncompress BZip files. 

bzcat Uncompress BZip data to standard output. 


compress Compress files with traditional Unix compression. 


uncompress Uncompress files with traditional Unix compression. 


zcat Uncompress to standard output (gzip or compress). 
zip Package and compress files in Windows Zip format. 
unzip Uncompress and unpack Windows Zip files. 
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7z Package and compress/uncompress 7-Zip files. 
munpack Extract MIME data to files. 

mpack Convert a file to MIME format. 

gpg Encrypt a file with the GNU Privacy Guard (GnuPG). 


Linux can pack and compress files in a variety of formats. 
The most popular formats are GNU Zip (gzip), whose com- 
pressed files are named with the .gz extension, and BZip, which 
uses the .bz2 extension. Other common formats include ZIP 
files from Windows systems (.zip extension), 7-Zip files (.7z 
and .Jzma extensions), and occasionally, classic Unix compres- 
sion (.Z extension). If you come across a format I don't cover, 
such as macOS sit files, ARC, Zoo, RAR, and others, learn more 
at https://oreil.ly/vO7I8. 


TIP 


Several popular commands have been adapted to work 
directly on compressed files. Check out zgrep, zless, 
zcmp, and zdiff, which work just like grep, less, cmp, 
and diff, respectively, but accept compressed files as argu- 
ments. Compare compressed and uncompressed files: 


> zdiff sample1.gz sample2 


An operation related to compression is the encoding of binary 
files as text for email attachments. Nowadays, most email cli- 
ents do this automatically, but I cover the commands munpack 
and mpack, which decode and encode on the command line, 
respectively. Finally, I cover the most popular file encryption 
command for Linux, gpg, the GNU Privacy Guard. 
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tar stdin stdout -file --opt --help  --version 
tar [options] [files] 


The tar command packs many files and directories into a 
single file for easy transport, optionally compressed. (It was 
originally for backing up files onto a tape drive; its name is 
short for “tape archiver?) TAR files are the most common 
file-packaging format for Linux. 


> tar -czf myarchive.tar.gz mydir Create archive 
> ls -1G myarchive.tar.gz 

-rw-r--r-- 1 smith 350 Nov 7 14:09 myarchive.tar.gz 
> tar -tf myarchive.tar.gz List contents 
mydir/ 

mydir/dir/ 

mydir/dir/file10 

mydir/filet 

mydir/file2 


> tar -xf myarchive.tar.gz Extract 


If you specify files on the command line, only those files are 
processed: 


> tar -xvf myarchive.tar.gz mydir/file3 mydir/file7 


Otherwise, the entire archive is processed. 


Useful options 


-c Create an archive from files and directories listed as arguments. 
-r Append files to an existing archive. 

-u Append new/changed files to an existing archive. 

-A Append one archive to the end of another: for example, tar 


-A -f first.tar second. tar appends the contents of 
second.tar to first.tar. Does not work for compressed archives. 


-t List (test) the archive. 


-x Extract files from the archive. 
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-C dir _ Extract the files into directory dir. 


-f file Read the archive from, or write the archive to, the given file. This is 
usually a TAR file on disk (such as myarchive.tar) but can also be a 
tape device (such as /dev/tape). 


-d Diff (compare) the archive against the filesystem. 
-Z Use gzip compression. 
-j Use bzip2 compression. 
-Z Use classic Unix compression. 
-V Verbose mode: print extra information. 
-h Follow symbolic links rather than merely copying them. 
-p When extracting files, restore their original permissions and 
ownership. 
g zip stdin stdout -file --opt --help  --version 


gzip [options] [files] 
gunzip [options] [files] 
zcat [options] [files] 


gzip and gunzip compress and uncompress files in GNU Zip 
format. Compressed files have the extension .gz. 


Sample commands 

gzip file Compress file to create file.gz. Original file is 
deleted. 

gunzip file.gz Uncompress file.gz to create file. Original file.gz is 
deleted. 

gunzip -c file.gz Uncompress a file to standard output. 

zcat file.gz Uncompress a file to standard output. 

cat file | gzip | .. Compress data ina pipeline. 
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cat file.gz | gunzip Uncompress data from a pipeline. 


tar -czf tarfile dir Pack directory dir into a gzipped TAR file. Add -v 
to print filenames as they are processed. 


bzip2 stdin stdout -file --opt --help  --version 


bzip2 [options] [files] 

bunzip2 [options] [files] 

bzcat [options] [files] 

bzip2 and bunzip2 compress and uncompress files in Bur- 


rows-Wheeler format. Compressed files have the exten- 
sion .bz2. 


Sample commands 
bzip2 file Compress file to create file.bz2. Original file is 
deleted. 
bunzip2 file. bz2 Uncompress file.bz2 to create file. Original 
file.bz2 is deleted. 
bunzip2 -c file.bz2 Uncompress a file to standard output. 
cat file | bunzip2 | .. Compress data ina pipeline. 


cat file.bz2 | bunzip2 Uncompress data from a pipeline. 
bzcat file.bz2 Uncompress a file to standard output. 


tar -cjf tarfile dir  Packdirectory dirinto a BZipped TAR file. Add 
-v to print filenames as they are processed. 
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compress stdin stdout -file -help — --version 


compress [options] [files] 

uncompress [options] [files] 

zcat [options] [files] 

compress and uncompress compress and uncompress files in 


classic Unix compression format (Lempel-Ziv). Compressed 
files have the extension .Z. 


Sample commands 


compress file Compress file to create file.Z. Original file is 
deleted. 

uncompress file.Z Uncompress file.Z to create file. Original file.Z 
is deleted. 

uncompress -c file.Z Uncompress a file to standard output. 

zcat file.Z Uncompress a file to standard output. 

cat file | compress | ... Compress data ina pipeline. 


cat file.Z | uncompress Uncompress data from a pipeline. 


tar -cZf tarfile dir Pack directory dir into a compressed TAR file. 
Use - cvZf to print filenames as they are 
processed. 


zip stdin stdout -help = --version 


zip archive.zip [options] [files] 


unzip [options] archive.zip [files] 


zip packs and compresses files in Windows Zip format, and 
unzip extracts them. Compressed files have the extension .zip. 


zip archive.zip filet file2... Pack. 
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zip -r archive.zip dir Pack a directory recursively. 


unzip -l archive.zip List contents. 
unzip archive.zip Unpack. 
72 --opt --help 


7z [command] [options] archive_file [(files | dirs)] 


The 7z command, also known as 7-Zip, packs and compresses 
files and directories. By default, it produces archive files in 
LZMA format with the filename extension .7z. It also supports 
other compression methods, such as ZIP, gzip, and BZip2, but 
you might as well use the original Linux commands (zip, gzip, 
and bzip2, respectively), which are more commonly found on 
Linux machines. 7z also extracts from a variety of archive files, 
even Microsoft CAB files. See the manpage for a full list. 


7z a archive.7z filet file2... Pack. 


7z a archive.7z dir Pack a directory (same as for files). 
7z 1 archive.7z List contents. 

7z x archive.7z Unpack. 

munpa ck stdin stdout -- opt 


munpack [options] mail_file 


mpack [options] files 


Modern email programs handle attachments so seamlessly that 
we rarely think about the process behind the scenes. The com- 
mands munpack and mpack work directly with attachments on 
the command line. For example, if you have an email message 
in a file, messagefile, and it contains a JPEG image and a PDF 
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file as attachments, munpack can extract both attachments into 
files: 


> munpack messagefile 
beautiful.jpg (image/jpeg) 
researchpaper.pdf (application/pdf) 


Its partner program, mpack, does the opposite: it inserts files 
as attachments into a MIME-format file. Create the file attach- 
ment.mime containing a MIME-encoded image, photo.jpg: 


> mpack -o attachment.mime photo. jpg 
Subject: My example photo 


gpg stdin stdout -file --opt --help  --version 


gpg [options] [args] 


The gpg command encrypts and decrypts files, manipula- 
tes digital signatures, maintains encryption keys, and more. 
Its part of the encryption application GNU Privacy Guard 
(GnuPG). 


The simplest approach, symmetric encryption, uses the same 
password to encrypt and decrypt a file (so don’t lose it!): 


> ls secret* 


secret 
> gpg -c secret 
Passphrase: XXXXXXXX Invent a password on the spot 


Repeat: XXXXXXXX 
> ls secret* 
secret secret.gpg Creates the encrypted file 


More common is public key encryption, which requires a pair 
of keys (public and private), encrypting with the public key, 
and decrypting with the private. Get started by creating a key 
pair, assuming your email address is srnith@example.com: 


> gpg --quick-generate-key smith@example.com \ 
default default never 
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Encrypt a file with public key encryption, creating secret.gpg: 


> gpg -e secret Use your default public key 
> gpg -e -r key secret Use a specific public key 


Encrypt and digitally sign a file, creating secret.gpg: 


> gpg -es secret 
Passphrase: xXxXXXXXxX 


Decrypt the file secret.gpg and verify any signature. (If you're 
not reprompted for a passphrase, gpg has cached it for now.) 


> rm secret Delete the original file 

> gpg secret.gpg Decrypt 

Passphrase: XXXXXXXX 

gpg: encrypted with 4096-bit ELG key, ID 3EE49F4396C9, 
created 2023-02-26 "John Smith <smith@example.com>" 


Good signature from "John Smith <smith@example.com>" 
> ls secret* 
secret secret.gpg See the original file, decrypted 


List the keys on your GnuPG keyring: 
> gpg --list-public-keys 
> gpg --list-secret-keys 
Useful options 
gpg has about 100 options. Here are some common ones. 
-r name Encrypt fora recipient on your public keyring. The name may be the 
key ID, email address, or various other parts of the key name. 
-u name Signas the user with this name from your keyring. 
-o file Write output to the given file. 


-a Create output in ASCII armor format instead of OpenPGP format. 
ASCII armor is plain text and suitable for inserting into email 
messages. Output files have the extension .asc rather than .gpg. 


-q Be quiet—don't print messages while running. 


-V Print more verbose messages while running. 
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Comparing Files 

diff _Line-by-line comparison of two files or directories. 
comm _Line-by-line comparison of two sorted files. 

cmp Byte-by-byte comparison of two files. 

shasum Compute checksums of the given files. 


md5sum Compute checksums of the given files (insecure). 
You can compare Linux files in at least three ways: 


Line by line (diff, comm), best suited to text files. 
e Byte by byte (cmp), often used for binary files. 


e By comparing checksums (shasum). Avoid older, weaker 
commands such as sum, cksum, and md5sum, which use inse- 
cure algorithms. 


diff stdout -file --opt --help  --version 
diff [options] file1 file2 


The diff command compares two files line by line, or two 
directories file by file. If there are no differences, diff produces 
no output. For text files, diff produces detailed reports of their 
differences. For binary files, diff merely reports whether they 
differ or not. 


The traditional output format looks like this: 


Affected line numbers, and the type of change 
< Corresponding section of file1, if any 


> Corresponding section of file2, if any 


For example, start with a file fileA: 
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Hello, this is a wonderful file. 
The quick brown fox jumped over 
the lazy dogs. 

Goodbye for now. 


Suppose you delete the first line, change “brown” to “blue” on 
the second line, and add a final line, creating a file fileB: 


The quick blue fox jumped over 
the lazy dogs. 

Goodbye for now. 

Linux r001z! 


The diff command reports two differences labeled 1,2c1 and 
4a4: 


> diff fileA fileB 


1,2¢1 fileA lines 1-2 became fileB line 1 
< Hello, this is a wonderful file. Lines 1-2 of fileA 
< The quick brown fox jumped over 

--- diff separator 

> The quick blue fox jumped over Line 1 of fileB 
4a4 Line 4 was added in fileB 

> Linux r00lz! The added line 


The leading symbols < and > are arrows indicating fileA and 
fileB, respectively. This output format is the default: many oth- 
ers are available, some of which can be fed directly to other 
tools. Try them out to see what they look like. 


Option Output format 


-¢ Context diff format, as used by the patch command (man 
patch). 

-D macro C preprocessor format, using #ifdef macro... #else... 
#Hendif. 

-u Unified format, which merges the files and prepends “-” for 


deletion and “+” for addition. Used by git. 
-y Side-by-side format; use -W to adjust the width of the output. 
-q Don’t report changes, just say whether the files differ. 
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diff can also compare directories. By default, it compares any 
same-named files in those directories and also lists files that 
appear in one directory but not the other: 


> diff dir1 dir2 


To compare entire directory trees recursively, run diff -r 
which produces a (potentially massive) report of all differences. 


> diff -r dir1 dir2 
Useful options 

-b_ Don't consider whitespace. 

-B_ Don’t consider blank lines. 

-i Ignore case. 


-r When comparing directories, recurse into subdirectories. 


diff is one member of a family of commands that operate on 
file differences. Some others are diff3, which compares three 
files at a time, and sdiff, which merges the differences between 
two files to create a third file according to your instructions. 


comm stdout -file --opt --help  --version 
comm [options] file1 file2 


The comm command compares two sorted files and produces 
three columns of output, separated by tabs: 


1. All lines that appear in file1 but not in file2. 
2. All lines that appear in file2 but not in filel. 
3. All lines that appear in both files. 


For example, if commfile1 and commfile2 contain these lines: 
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commfile1: commfile2: 


apple baker 
baker charlie 
charlie dark 


then comm produces this three-column output: 


> comm commfile1 commfile2 
apple 
baker 
charlie 
dark 


Useful options 


-1 Suppress column 1. 

-2 Suppress column 2. 

-3 Suppress column 3. 

-23 Show lines that appear only in the first file. 
-13 Show lines that appear only in the second file. 


-12 Show only common lines. 


cmp stdin stdout -file --opt --help  --version 
cmp [options] file1 file2 [offset1 [offset2]] 


The cmp command compares two files. If their contents are the 
same, cmp reports nothing; otherwise, it lists the location of the 
first difference: 


> cmp myfile yourfile 
myfile yourfile differ: byte 225, line 4 


By default, cmp does not tell you what the difference is, only 
where it is. It also is perfectly suitable for comparing binary 
files, as opposed to diff, which operates best on text files. 
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Normally, cmp starts its comparison at the beginning of each 
file, but if you provide offsets, it starts elsewhere: 


> cmp myfile yourfile 10 20 


This comparison begins at the 10th character of myfile and the 
20th of yourfile. 


Useful options 


-1 Long output: print all differences, byte by byte: 
> cmp -l myfile yourfile 
225 167 127 


The output says that at offset 225 (in decimal), myfile has a small “w” (octal 
167) but yourfile has a capital “W” (octal 127). 


-s Silent output: don’t print anything, just exit with an appropriate return code; 
0 if the files match, 1 if they don’t. (Or other codes if the comparison fails.) 


shasum stdin stdout -file --opt --help  --version 


shasum -a (256|384|512|512224|512256) [options] files 


shasum --check file 


The shasum command calculates and validates checksums to 
verify that files are unchanged. By default, shasum uses an 
insecure algorithm called SHA-1. Add the option -a 256 (or 
a higher value) for cryptographically secure results. Here I 
produce a 256-bit checksum of the given files (64 hexadecimal 
digits) using the secure SHA-256 algorithm: 


> shasum -a 256 myfile SHA-256 algorithm 
e8183aaa23aa9b74c7033cbc843041fcfid1e9e937... myfile 


The second form of the command tests whether a checksum 
matches the original file, using - - check: 


> shasum -a 256 myfile myfile2 myfile3 > /tmp/mysum 
> cat /tmp/mysum 
e8183aaa23aa9b74c7033cbc843041fcf1id1ie9e937... myfile 
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2254f6879ae5fdf174b3a2ebbdc7fb4fa4ieOddf4a... myfile2 
Obfa73d888300e3d4f5bc9ac302c1eb38e37499bS5e... myfile3 
> shasum --check /tmp/mysum 

myfile: OK 

myfile2: OK 

myfile3: OK 

> echo "new data" > myfile2 Change myfile2 

> shasum --check /tmp/mysum 

myfile: OK 

myfile2: FAILED 

myfile3: OK 

shasum: WARNING: 1 computed checksum did NOT match 


Two different files are unlikely to have the same SHA-256 
checksum, so comparing their checksums is a reliable way to 
detect if two files differ. 


> shasum -a 256 myfile | cut -c1-64 > /tmp/sum1i 
> shasum -a 256 myfile2 | cut -c1-64 > /tmp/sum2 
> diff -q /tmp/sum1 /tmp/sum2 

Files /tmp/sum1 and /tmp/sum2 differ 


md5sum stdin stdout -file --opt --help  --version 
md5sum files | --check file 


The md5sum command computes checksums using an insecure 
algorithm called MD5. Do not use it for production work. Run 
it much like shasum: 


> md5sum myfile myfile2 myfile3 > /tmp/mysum Generate 
> md5sum --check /tmp/mysum Check 


Converting Files to Other Formats 


pandoc Convert from one markup language to another. 
hxselect Extract information from an HTML file. 


jq Extract information from a JSON file. 
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xmllint — Validate and extract information from an XML file. 
csvtool — Extract information from a comma-separated values (CSV) file. 
split Split up a file simply into multiple files. 


csplit Split up a file into multiple files using complex criteria. 


Have you ever tediously converted a text file from one format 
to another by hand, or manually extracted values from CSV, 
JSON, or XML files? Never again! Linux has an arsenal of file- 
conversion commands that make this kind of work disappear. 


stdin stdout -file --opt --help  --version 
pandoc 
pandoc [options] [files] 


The amazing pandoc command converts files of many for- 
mats into many other formats. It handles HTML, JSON, CSV, 
LaTeX, Microsoft DOCX and PPTX, DocBook XML, man- 
pages, numerous flavors of markdown and wiki text, and doz- 
ens of other formats. The results aren't perfect and may require 
a bit of manual cleanup, but they’re surprisingly good.® 


Running pandoc couldn't be simpler: just provide an input file 
and pick an output format with the -t option, and pandoc 
prints the converted results. Look how easy it is to convert a 
CSV file to HTML, GitHub markdown, LaTeX, and JSON: 


> cat data.csv Original file 


one, two, three, four, five,six,seven 
ONE, TWO, THREE, FOUR, FIVE, SIX, SEVEN 


1,25354,;556,7 

> pandoc -t html data.csv CSV to HTML 
<table> 

<thead> 


6 Iran pandoc to convert material from the third edition of this book, 
written in DocBook XML, to AsciiDoc for the fourth edition. 
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<tr class="header"> 
<th>one</th> 
<th>two</th> 


> pandoc -t gfm data.csv CSV to markdown 
| one | two | three | four | five | six | seven | 


[wees Foes |------- ecees [pe oe>s [sees [peeneee | 
| ONE | TWO | THREE | FOUR | FIVE | SIX | SEVEN | 


}1 [2 | 3 | 4 | 5 16 |7 | 

> pandoc -t latex data.csv CSV to LaTeX 
\begin{longtable}[ ]{@{}LLLLLLL@{}} 

\toprule 


one & two & three & four & five & six & seven ... 
> pandoc -t json data.csv CSV to JSON 
{"blocks":[{"t":"Table","c":[[], ... 


Redirect the results to a file, or better yet, add the -o option 
to write the results to an output file, particularly if they’re in a 
binary format. If pandoc can guess the output format from the 
output filename, you can omit the -t option: 


> pandoc -o page.pdf page.html HTML to PDF 


If pandoc can't guess the format of the input file, add the option 
-f (“from”): 


> pandoc -o page.pdf -f html page.html 
Useful options 


pandoc has many more options than I list here, and it supports 
configuration files for setting multiple options conveniently. 
See the manpage for details. 

--list-input-formats _ Listall supported input formats (34 so far). 


--list-output-formats _ Listall supported output formats (57 so far). 


-f input_format Explicitly convert from this format. 
-t output_format Explicitly convert to this format. 
-o file Write the output to this file. 
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--wrap=type Determine whether text should be wrapped in 
the output. Values are none for no wrapping, 
auto to wrap at 72 columns, or preserve to 
keep the same wrapping as the original file. 


--columns=N Wrap the next at column A (default=72). 


hxselect stdin stdout 


hxselect [options] CSS_selectors 


The hxselect command extracts strings from HTML data 
based on CSS selectors. For example, extract all div tags from a 
file page.html: 


> cat page. html Original file 
<html> 
<head> 
</head> 
<body> 
<div> 
This is the first div. 
</div> 
<div class="secondary"> 
This is the second div. 
</div> 
</body> 
</html> 
> hxselect 'div' < page.html Print one div 
<div> 
This is the first div 
</div><div class="secondary"> 
This is the second div 
</div> 
> hxselect -c 'div' < page.html Contents 
This is the first div 
This is the second div 
> hxselect -c 'div.secondary' < page.html By selector 
This is the second div 
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For best results, pass the content first through hxnormalize -x 
to clean up the HTML code. 


> hxnormalize -x page.html | hxselect ... 
Pipe the source of a web page to hxselect using curl: 


> curl url | hxnormalize -x | hxselect ... 


Useful options 


-c Just print the content within tags, not the tags themselves. 


-i Match strings case-insensitively. 


jq stdin stdout -file --opt --help  --version 
jq [options] filter [JSON_files] 


The jq command extracts and manipulates JSON data, accord- 
ing to a filter that you provide, and pretty-prints the results. 
I present a few examples of using this powerful tool (see the 
manpage for more). 


> cat book. json Original file 
{ 

"title": "Linux Pocket Guide", 

"author": "Daniel J. Barrett", 


"publisher": { 
"name": "O'Reilly Media", 
"url": "https://oreilly.com" 
}, 
"editions": [1, 2, 3, 4] 


} 

> jq .title book.json Simple value 
"Linux Pocket Guide" 

> jq .title,.author book. json Multiple values 


"Linux Pocket Guide" 
"Daniel J. Barrett" 
> jq .publisher book.json Object 


{ 
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"name": "O'Reilly Media", 
"url": "https://oreilly.com" 


} 
> jq .publisher.url book. json Nested value 
"https://oreilly.com" 
> jq .editions book. json Array 
[ 

13 

2, 

3, 

4 
] 
> jq .editions[0] book. json One array value 
1 
> jq '.editions|length' book.json Array length 
4 
> jq '.editions|add' book. json Sum 1+2+3+4 
10 
> cat oneline.json A one-line JSON file 
{"title":"Linux Pocket Guide","author": ... 
> jq < oneline. json Pretty-print the file 
{ 

"title": "Linux Pocket Guide", 


"author": ... 


Useful options 


-f file Read the filter from a file instead of the command line. 


-S Sort the output by its JSON keys. 


xmlilint stdin stdout -file --help — --version 
xmllint [options] [XML_files] 


The xmllint command validates and extracts XML data. Vali- 
dating a valid XML file just prints the contents. Add the option 
--noout to suppress the output. 
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> xmllint good.xml 

<?xml version="1.0"?> 

<hello> </hello> 

> xmllint --noout good.xml 

> echo $? 

0 Success code 


An invalid XML file prints an error: 


> cat bad.xml 

<?xml version="1.0"?> 

<hello> </helo> Mismatched tags 
> xmllint bad.xml 

bad.xml:2: parser error : Opening and ending tag 
mismatch: hello line 2 and helo 


> echo $? 
1 Error code 


Provide an XPath expression to extract data: 


> cat book.xml Original file 
<?xml verston="1.0"?> 
<book> 
<title>Linux Pocket Guide</title> 
<author>Daniel J. Barrett</author> 
<pub> 
<name>0'Reilly Media</name> 
<url>https://oreilly.com</url> 
</pub> 
<eds> 
<ed id="1">First edition</ed> 
<ed id="2">Second edition</ed> 
<ed id="3">Third edition</ed> 
<ed id="4">Fourth edition</ed> 
</eds> 
</book> 
> xmllint --xpath '//book/title' book.xml 
<title>Linux Pocket Guide</title> 
> xmllint --xpath '//book/title/text()' book.xml 
Linux Pocket Guide 
> xmllint --xpath '//book/pub/url/text()' book.xml 
https://oreilly.com 
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> xmllint --xpath '//book/eds/ed[@id][4]' book.xml 

<ed id="4">Fourth edition</ed> 

> xmllint --xpath '//book/eds/ed[@id][4]/text()' \ 
book.xml 

Fourth edition 


Useful options 


--xpath path Query the given XPath expression in the XML data. 


--format Print the output nicely formatted. 
--noout Don’t print output. 
csvtool stdin stdout -file --opt --help  --version 


csvtool [options] command [command_args] CSV_files 


The csvtool command extracts data from CSV files. It can 
extract columns: 


> cat data.csv Original file 
one, two, three, four, five,six, seven 

ONE, TWO, THREE, FOUR, FIVE, SIX,SEVEN 
1,2,3,4,5,6,7 

> csvtool col 3 data.csv One column 
three 

THREE 

3 

> csvtool col 2,5-7 data.csv Multiple columns 
two, five,six,seven 

TWO, FIVE, SIX, SEVEN 

2,5,6,7 


It can count the number of rows and columns: 


> csvtool height data.csv Number of rows 

3 

> csvtool width data.csv Number of columns (max) 
o 


It can insert values from each row into longer strings: 
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> csvtool format 'Third column is "%3"\n' data.csv 
Third column is "three" 

Third column is "THREE" 

Third column is "3" 


It can remove the first row, which often contains headings: 


> csvtool drop 1 data.csv 
ONE, TWO, THREE, FOUR, FIVE, SIX, SEVEN 
1,2;3;4,5;6,7 


It can isolate a single value, say, row 2, column 6: 


> csvtool drop 1 data.csv \ Delete the first row 
| csvtool head 1 - \ Print the first remaining row 
| csvtool col 6 - Extract column 6 

SIX 


and much more. The csvtool manpage is sparse, so run 


csvtool --help for more information. 


Useful options 


-c char Change the input separator character from a comma to char. For tab 
characters, use -c TAB. 


-0 Same as -c but for the output separator character. 


split stdin stdout -file --opt --help  --version 
split [options] [file [prefix]] 


The split command divides a file into equal-sized pieces (for 
various definitions of “equal-sized”) and stores them in sepa- 
rate files. The output files are named sequentially, such as Xaa, 
Xab, Xac, and so on, by default. X is called the prefix and the 
endings aa, ab, and so on, are the suffix. 


Divide the input file by number of lines: 


> split hugefile 1000 lines each 
> split -1l 2000 hugefile 2000 lines each 
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or by bytes: 

> split -b 1024 hugefile 1024 bytes each 
or split it at a given character, called the separator: 

> split -t@ -11 hugefile Separator character is @ 
Optionally change the prefix with a final argument: 


> split hugefile part- Set the prefix to “part-” 
> ls 
hugefile part-aa part-ab part-ac part-ad 


split can help you work with files that are too large to load 
into other programs, or too large to store on offline media. 
Reassemble the parts afterward with cat: 


> cat part-* > /tmp/outfile Join part-aa, part-ab, etc. 


Useful options 


-1 lines Split into files of this many lines or records. 

-b bytes Split into files of this many bytes. 

-t separator Set the separator character, if any, between records. 
-a length Set the length of the suffix, in characters. 


-d Use numeric suffixes instead of letters, starting from 
zero. Change the starting value to N with - -numeric- 
suffixes N. 


csplit stdout -file --opt --help  --version 
csplit [options] file patterns 


The csplit command divides one file into many based on 
regular expressions. The output files are named sequentially, 
such as xx01, xx02, xx03, and so on, by default. xx is called 
the prefix and the endings 01, 02, and so on, are the suffix. By 
default, csplit prints the size of each file it creates. 
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The syntax is a bit unusual. After the input filename, provide 
one or more patterns that represent dividing lines in the file. 
Patterns have two types: 


Regular expressions 
An expression (see Table 2-2) enclosed in forward slashes, 
such as /4[A-Z]*$/ for a line of all uppercase letters. Each 
expression is matched just once by default. 


Repeaters 
A repeater has the form '{N}' which means “match the 
preceding pattern up to N times.” An example is '{2}' 
which matches its preceding pattern twice. The special 
repeater '{*}' matches its preceding pattern as many 
times as possible until the input ends. 


csplit is great for splitting up structured text such as HTML, 
XML, or programming source code. 


> cat page.html Original file 
<html> 
<head> 
</head> 
<body> 
<div> 
This is the first div. 
</div> 
<div class="secondary"> 
This is the second div. 
</div> 
</body> 
</html> 


Use four patterns (in quotes below) to split a file at the first 
<head>, then at the next <body>, and then at every <div> tag: 


> csplit page.html '/<head>/' '/<body>/' '/<div/' '{*}' 
7 

19 

9 Sizes of the output files, in bytes 

49 

86 
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View the five output files: 


> 1s xx* 


xx0Q@ xx01 xx02 xx03 xx04 


> cat xx00 
<html> 
> cat xx01 
<head> 
</head> 
> cat xx02 
<body> 
> cat xx03 
<div> 


This is the first div 


</div> 
> cat xx04 


<div class="secondary"> 
This is the second div 


</div> 
</body> 
</html> 


Useful options 


-f prefix Set the prefix for output filenames. 


-n Length Set the suffix for output filenames to be /ength digits long. 


-S Silent operation: don’t print the sizes of the generated files. 


PDF and PostScript File Handling 


pdftotext 
ps2ascit 
pdfseparate 
pdftk 

pdf2ps, ps2pdf 


ocrmypdf 


Extract text from PDF files. 
Extract text from PostScript or PDF files. 
Extract individual pages from a PDF file. 


Split, join, rotate, and otherwise manipulate PDF files. 


Convert between PDF and PostScript file formats. 


Perform optical character recognition (OCR) on a PDF. 


PDF and PostScript File Handling 
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To view PDF files and PostScript files on Linux, you'll need a 
graphical desktop and a document viewer like these:’ 


> okular sample. pdf KDE's document viewer 
> evince sample. pdf GNOME 's document viewer 
> gv sample.pdf Ghostscript viewer 


In addition, Linux has a rich set of command-line tools to work 
with PDF and Postscript files without displaying them. They 
are well worth learning, especially the amazing pdftk. 


pdftotext -file --opt help 


pdftotext [options] [file.pdf [outfile.txt]] 


The pdftotext command extracts text from a PDF file and 
writes it to a file. This works if the PDF contains actual text, not 
images that look like text (in which case, run ocrmypdf first). 


> pdftotext sample. pdf Creates sample.txt 


Useful options 


-f N Begin with page NV of the PDF file. You must have a space between 
the option and the number. 

-LN End with page N of the PDF file. You must have a space between 
the option and the number. 


-htmlmeta Generate HTML rather than plain text (creates sample.html). 


-eol OS Write end-of-line characters for the given operating system, 0S, 
which can be dos, mac, or unix. 


7 To edit PDF files, the best Linux program I’ve used is Master PDF 
Editor by Code Industry, a commercial product. 
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ps2ascii stdin stdout -file --version 
ps2ascii file.(ps|pdf) [outfile.txt] 


The ps2ascii command extracts text from a PostScript file. 
It’s a simple command with no options.® To extract text from 
sample.ps and place it into /tmp/extracted. txt: 


> ps2asciit sample.ps /tmp/extracted.txt 


ps2ascii can also extract text from a PDF file, though you 
wouldn't guess that from the command name. 


> ps2ascit sample.pdf /tmp/extracted2.txt 


pdfseparate ~opt ~help 
pdfseparate [options] [file.pdf] [pattern.txt] 


The pdfseparate command splits a PDF file into separate PDF 
files, one per page. For example, if one.pdf has 10 pages, then 
the following command creates 10 PDF files in /tmp named 
split1.pdf through split10.pdf, each containing one page: 

> pdfseparate one.pdf /tmp/split%d. pdf 

The final argument is a pattern to form the names of the 


individual page files. The special notation %d stands for the 
extracted page number. 


8 Actually, ps2ascii --help describes command-line options, but they 
dont work. They are the options of a related command, gs, invoked by 
ps2ascii. 
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Useful options 


-f N_ Begin with page N of the PDF file. You must have a space between the 


option and the number. 
-lL NW End with page W of the PDF file. You must have a space between the 
option and the number. 
pdftk stdin stdout --help — --version 


pdftk [arguments] 


pdftk is the Swiss Army knife of PDF commands. This versatile 
program can extract pages from a PDF file, join several PDFs 
into one, rotate pages, add watermarks, encrypt and decrypt 
files, and much more, all from the command line. This power 
comes with quirky syntax, unfortunately, but with a little effort 
you can learn a few useful tricks. 


To join the files one.pdf and two.pdf into a single PDF file, 
combined. pdf: 


> pdftk one.pdf two.pdf cat output combined. pdf 


To extract pages 3, 5, and 8-10 from the file one.pdf and write 
them to new.pdf: 


> pdftk one.pdf cat 3 5 8-10 output new.pdf 


Extract the first five pages from one.pdf and the odd-numbered 
pages from two.pdf and combine them as combined. pdf. 


> pdftk A=one.pdf B=two.pdf cat A1-5 Bodd output \ 
combined. pdf 


Copy the file one.pdf to new.pdf, but with page 7 rotated by 90 
degrees clockwise (“east”): 


> pdftk one.pdf cat 1-6 7east 8-end output new.pdf 


Interleave the pages of one.pdf and two.pdf, creating mixed.pdf. 
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> pdftk one.pdf two.pdf shuffle output mixed. pdf 


The criteria for page selection are very powerful and typically 
appear before the output keyword. They consist of one or more 
page ranges with qualifiers. A page range can be a single page 
like 5, a range like 5-10, or a reverse range like 10-5 (which 
reverses the pages in the output). Qualifiers can remove pages 
from a range, like 1-100~20-25, which means “all pages from 
1 to 100 except for pages 20 to 25.” They can also specify only 
odd pages or even pages, using the keywords odd or even, and 
rotations using the compass directions north, south, east, and 
west. I’ve only scratched the surface of pdftk’s abilities. The 
manpage has more details and examples. 


pdf2ps - file ~-help 
pdf2ps [options] file.pdf [file.ps] 
ps2pdf [options] file.ps [file.pdf] 


The pdf2ps command converts an Adobe PDF file into a Post- 
Script file. If you don't provide an output filename, the default 
is to use the input filename, with .pdf replaced by .ps. 


> pdf2ps sample.pdf converted.ps 


To go in the opposite direction, converting a PostScript file to 
PDF format, use ps2pdf: 


> ps2pdf sample.ps converted. pdf 


ocrmypdf -file --opt --help — --version 
ocrmypdf [options] input_file output_file 


The ocrmypdf command performs optical character recognition 
(OCR) to create a searchable PDF file. The input file can be an 
image file or a PDF file that contains images. 
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> ocrmypdf imageoftext.png outfile. pdf Convert 
> okular outfile. pdf Display 


Useful options 


-l language Use language rather than English. For a list of languages, run 
tesseract --list-langs. 


-r Attempt to rotate pages into their correct orientation. 


Printing 

lpr Printa file. 

lpq _ View the print queue. 

lprm Remove a print job from the queue. 

Linux has two popular printing systems, CUPS and LPRng. 
Both systems use commands named lpr, lpq, and lprm, but 


their options are different on CUPS and LPRng. I'll present 
common options that work with both systems. 


To install and maintain printers, GNOME and KDE have 
printer configuration tools in their system settings. To trouble- 
shoot a CUPS printer, visit http://localhost:631 to access your 
computer’s CUPS management system. 


Ipr stdin ~-help 
lpr [options] [files] 
The lpr (line printer) command sends a file to a printer: 


> lpr myfile Print on default printer 
> lpr -P myprinter myfile Print on a named printer 
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Useful options 


-P printername Send the file to printer printername, which you have 


previously set up. 

-# N Print N copies of the file. (The option is a literal hash 
mark.) 

-J name Set the job name that prints on the cover page (if your 


system is set up to print cover pages). 


Ipq stdout ~-help 
lpq [options] 


The lpq (line printer queue) command lists print jobs that are 
waiting to be printed. 


Useful options 


-P printername List the queue for printer printername. 


-a List the queue for all printers. 
-1 Be verbose: display information in a longer format. 
Iprm stdout 


lprm [options] [job_IDs] 


The Lprm (line printer remove) command cancels one or more 
print jobs. Use lpq to learn the ID of the desired print jobs (say, 
61 and 78), then run: 


> Uprm -P printer_name 61 78 


If you don't supply any job IDs, your current print job is can- 
celed. (The superuser can cancel other users’ jobs.) The -P 
option specifies which print queue contains the job. 
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Spellchecking 


look Look up the spelling of a word quickly. 
aspell Interactive spelling checker. 
spell Batch spelling checker. 
Linux has several spellcheckers built in. If youre accustomed to 


graphical spellcheckers, you might find Linux’s text-based ones 
fairly primitive, but they're useful in a pinch (or a pipeline). 


look stdout --opt --help — --version 
look [options] prefix [dictionary_file] 


The look command prints words that begin with a given string 
prefix. The words are located in a dictionary file (default /usr/ 
share/dict/words): 


> look bigg 
bigger 
biggest 
Biggs 


If you supply your own dictionary file—any text file with 
alphabetically sorted lines—look prints all lines in the dictio- 
nary that begin with the given prefix. 


Useful options 


-f Ignore case. Needed only if you supply a dictionary file. 
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aspell stdin stdout -file --opt --help  --version 
aspell [options] file 


aspelLl is an interactive spellchecker. It identifies words that it 
doesn't recognize and presents alternatives. A few useful com- 
mands are: 


aspell -c file 
Interactively check, and optionally correct, the spelling of 
all words in file. 


aspell dump master 
Print aspell’s master dictionary on standard output. 


aspell help 
Print a concise help message. See http://aspell.net for more 
information. 


spell stdin stdout -file --opt --help  --version 
spell [files] 


The spell command prints all words in the given files that are 
misspelled, according to its dictionary. It is not interactive. 


> cat badwords 

This Linux file has some spelling errors. 

You may naturaly wonder if a spelling checker 
will pick them up. Careful Linuxx users should 
run thier favorite spelling checker on this file. 
> spell badwords 

naturaly 

Linuxx 

thier 
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CHAPTER 3 
System Administration Basics 


Becoming the Superuser 


Normal users, for the most part, can modify only the files they 
own. One special user, called the superuser or root, has full 
access to the machine and can do anything on it. Superuser 
privileges are mainly for system administration tasks; use them 
only when absolutely necessary so you don’t accidentally harm 
your Linux system. And don't log in as root unless you must 
(e.g., when rescuing a broken boot process). 


WARNING 


Superuser commands can destroy a Linux system if you’re 
not careful. 


You can become the superuser in several ways. One is to use 
the sudo command to gain superuser abilities for the duration 
of a single command. Simply type “sudo” followed by the com- 
mand. You may be prompted for your password, depending on 
how sudo is configured on your machine: 
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> sudo rm some_protected_file 
[sudo] password: xxxxXxxxx Your own password 


To maintain your superuser powers without constantly run- 
ning sudo, launch a superuser shell with either of the following 
commands: 


> sudo -s 
> sudo bash 


A superuser shell is convenient, say, for browsing through 
many protected directories with cd. When finished executing 
commands as the superuser, press *D or run exit to end 
the superuser shell and become yourself again. If you forget 
whether your shell is a superuser shell or just a normal one, 
check your identity with the whoami command. If you're the 
superuser, it displays root. 


Another way to become the superuser is the su command, 
which also creates a superuser shell, but you'll need a differ- 
ent password, called the root password, to use it. (Whoever 
installed Linux chose the root password during installation.) 


> su 
Password: XXXXXXXX root password 
# 


Your shell prompt may change, often to a hash mark (#), to 
indicate you are the superuser. Add the option -1 to run a login 
shell that includes root’s full environment, such as root’s shell 
aliases. 


> su -l Run a login shell 
Password: XXXXXXXX root password 
# 


If you provide a username to sudo or su, you become that user, 
provided you have the prerequisites: 


> sudo -u sophia command Become user sophia with sudo 
[sudo] password: xxxxxxxx You must have sudo permission 

> su sophia Become user sophia with su 
Password: XXXXXXXX You must know sophia's password 
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Use sudo rather than su whenever possible, especially if your 
system has multiple users. su relies on a shared password, 
which is a security concern. sudo uses your own password, 
but it must be configured to do so (and many distros come 
with sudo preconfigured). sudo also provides precise control 
over privileges in the file /etc/sudoers, and it even logs the 
commands that users run. A full discussion is beyond the scope 
of this book: see man sudo for details. 


Viewing Processes 


ps List processes. 

pgrep List the IDs of processes that match a regular expression. 
uptime View the system load. 

w List active processes for all users. 

top Monitor resource-intensive processes interactively. 


free Display free memory. 


A process is a unit of work on a Linux system. Each program 
you run represents one or more processes, and Linux provides 
commands for viewing and manipulating them. Every process 
is identified by a numeric process ID, or PID, and can be exam- 
ined in the directory /proc (see “Kernel-Related Directories” on 
page 33). 


Processes are different from jobs (see “Shell Job Control” on 
page 49). Processes are part of the OS. Jobs are higher-level 
constructs known only to the shell in which they’re running. 
A running program comprises one or more processes; a shell 
job consists of one or more programs executed as a shell 
command. 
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ps stdout -help = --version 
ps [options] 


The ps command displays information about your running 
processes, and optionally the processes of other users: 


> ps 
PID TTY TIME CMD 
4706 pts/2 00:00:01 bash 
15007 pts/2 00:00:00 emacs 
16729 pts/2 00:00:00 ps 


ps has at least 80 options; I cover just a few useful combina- 
tions. If the options seem arbitrary or inconsistent, it’s because 
the supplied ps command (GNU ps) incorporates the features 
of several other, competing ps commands, attempting to be 
compatible with all of them. 


View your processes: 
> ps -uXx 
View all processes owned by user “smith”: 
> ps -U smith 
View all occurrences of a running program: 
> ps -C python 
View processes on terminal N: 
> ps -tW 
View particular processes 1, 2, and 3505: 
> ps -p1,2,3505 


View all processes with their command lines truncated to the 
width of the screen: 


> ps -ef 
View all processes with full command lines: 


> ps -efww 
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View all processes in a threaded view that indents child pro- 
cesses below their parents: 

> ps -efH 

Use grep and other filter commands to extract information 


more finely from the output of ps: 


> ps -ux | grep python 


pgrep stdout -help — --version 
pgrep [options] regex 


The pgrep command prints the process IDs (PIDs) of pro- 
cesses that match certain criteria, given by a regular expression. 
(A similar command, pidof, matches only fixed strings.) For 
example, print the PIDs of all running python processes: 


> pgrep python 
4675 

79493 

82866 

83114 


or count the processes by adding the option -c: 


> pgrep -c python 
4 


Limit the results to processes owned by a particular user: 


> pgrep -u smith 
79493 


pgrep is most useful for passing a list of related PIDs to another 
command. For example, locate processes that match a string, 
using ps and command substitution (see “Command substitu- 
tion” on page 45). 


> ps -fwp $(pgrep python) 
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See more examples of pgrep with the kill command in “Con- 
trolling Processes” on page 169. 


Useful options 


-d string Use string as the delimiter character between PIDs (newline by 
default). 


-u user __ Printonly processes running as the given username or user ID 
(UID). This is the effective user ID. For real user ID, use -U. 


-f Match against the process's full command line, not just its name. 
-x Exact match, rather than a substring match. 
-v Print PIDs of processes that don’t match. 
uptime stdout --help — --version 
uptime 


The uptime command tells you how long the system has been 
running since the last boot: 


> uptime 
10:54pm up 8 days, 3:44, 3 users, 
load average: 0.89, 1.00, 2.15 


This information is, from beginning to end: the current time 
(10:54 p.m.), system uptime (8 days, 3 hours, 44 minutes), 
number of users logged in (3), and system load average for 
three time periods: 1 minute (0.89), 5 minutes (1.00), and 15 
minutes (2.15). The load average is the average number of 
processes ready to run in that time interval. 
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Ww stdout --opt --help — --version 
w [username] 

The w command displays the current process running in each 
shell for all logged-in users: 


> Ww 

10:51pm up 8 days, 3:42, 8 users, 

load average: 2.02, 3.79, 5.44 

USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT 
barrett pts/O :0 Sat 2pm 27:13m 0.07s 0.07s emacs 
jones pts/1 host1 6Sep03 2:33m 0.74s @.21s bash 
smith pts/2 host2 6Sep03 0.00s 13.35s 0.045 w 


The top line is the same one printed by uptime. The columns 
indicate the user’s terminal, originating host or X display (if 
applicable), login time, idle time, two measures of the CPU 
time (run man w for details), and the current process. Provide a 
username to see only that user’s information. 


For the briefest output, try w -hfs. 
Useful options 


-h_ Don't print the header line. 
-f Don’t print the FROM column. 
-s Don't print the JCPU and PCPU columns. 


top stdout -help — --version 
top [options] 


The top command monitors the most active processes, updat- 
ing the display at regular intervals (every 3 seconds, by default): 
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> top 

top - 13:02:14 up 1 day, 32 min, 9 users, 

Tasks: 719 total, 2 running, 717 sleeping, 0 stopped... 
%Cpu(s): 0.3 us, 0.2 sy, 0.0 ni, 99.6 id, 0.0 wa, 

MiB Mem : 31950.5 total, 3040.6 free, 5267.4 used ... 
MiB Swap: 32000.0 total, 31991.7 free, 8.2 used ... 


PID USER PR NI VIRT SHR S %CPU %MEM TIME CMD 
26265 smith 20 0 1092 840 R 4.7 0.2 0:00 top 
1 root 20 0 540 472 S 0.0 0.1 0:07 systemd 
914 www 0 0 0 OSW 0.0 0.0 0:00 httpd 


While top runs, change its behavior interactively by pressing 
keys, such as setting the update speed (s key), hiding idle pro- 
cesses (i), or killing processes (k). Press h to list the keystrokes 
and q to quit. For similar commands to monitor your system's 
1/O and network bandwidth, try iotop and iftop. Also check 
out the all-in-one command btop, which displays stats for your 
CPUs, disks, processes, and network interfaces together. 


Useful options 


-nW_ Perform N updates, then quit. 
-dN Update the display every N seconds. 


-pN_ Display the process with PID NW. You may repeat this option for up to 20 
PIDs. 


-c _ Display the command-line arguments of processes. 


-b Print on standard output instead of behaving like an interactive 
application. Suitable for piping or redirecting the output to a file. To 
save a single iteration of top toa file, runtop -b -n1 > outfile. 


free stdout -help — --version 
free [options] 


The free command displays total memory use in KB: 
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> free 


total used free shared buf/cache available 
Mem: 523812 491944 31868 0 224812 299000 
Swap: 530104 © 530104 


The Linux kernel reserves memory for caching purposes (buf/ 
cache column), so your best estimate of free RAM in the pre- 
ceding output is in the available column (ie., 299,000 KB) 
rather than the free column (31,868 KB). 


Useful options 


-s N Runcontinuously and update the display every NV seconds. 
-m _ Display amounts in MB. 
-h Display amounts in human-readable units like “Gi” for GB. 


-t Add a totals row at the bottom. 


Controlling Processes 


kill Terminate a process (or send it a signal). 

pkill Terminate processes by name (or send them a signal). 

timeout Kill a command that runs for too long. 

nice Invoke a program at a particular priority. 

renice Change a process's priority as it runs. 

nohup Run a process that continues after you log out. 

flock _ Ensure that only one instance of a command runs at a time. 
Once processes are started, they can be stopped, restarted, kil- 
led, and prioritized. I discussed some of these operations as 


handled by the shell in “Shell Job Control” on page 49. Let's 
continue to killing and prioritizing processes. 
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kill stdout ~-help 
kill [options] [process_ids] 


The kill command sends a signal to a process. This can 
terminate a process (the default action), interrupt it, suspend 
it, crash it, and so on. You must own the process, or be the 
superuser, to affect it. To terminate process 13243, run: 


> kill 13243 


If this does not work—some programs catch this signal without 
terminating—add the -KILL or (equivalently) -9 option: 


> kill -KILL 13243 


which is virtually guaranteed to work. However, this is not a 
clean exit for the program, which may leave resources allocated 
(or cause other inconsistencies) upon its death. 


To learn the PID of a process, run ps and examine the output, 
or better yet, run pgrep to produce the PID directly: 


> ps -uax | grep emacs 


smith 8374 ... emacs myfile.txt Actual emacs process 
smith 9051 ... grep emacs Spurious result to ignore 
> pgrep emacs More accurate output 
8374 


Now kill this process by PID, or kill it by name using command 
substitution (see “Command substitution” on page 45): 


> kill 8374 Kill by PID 
> kill $(pgrep emacs) Kill by name 


Or use the pkill command to kill all processes for a given 
program: 


> pkill emacs 
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In addition to the kill program in the filesystem (usually /bin/ 
kill), most shells have built-in kill commands. Their syntax 
and behavior differ, but they all support the following usage: 


kill -N PID 
kill -NAME PID 


where N is a signal number, NAME is a signal name without 
its leading “SIG” (e.g., to send the SIGHUP signal, use -HUP), 
and PID is the ID of the process to kill. To see a complete 
list of signals transmitted by kill, run kill -1, though its 
output differs depending on which kill youre running. For 
descriptions of the signals, run man 7 signal. 


timeout help version 
timeout [options] seconds command... 


The timeout command sets a time limit for running a program, 
in seconds. If the program runs longer than the limit, timeout 
kills it. As a demonstration, here is a sleep command that 
should run for one minute but is killed after 3 seconds: 


> timeout 3 sleep 60 Killed after 3 seconds 


Asa more practical example, play music from your MP3 collec- 
tion for an hour (3,600 seconds), then stop: 


> timeout 3600 mplayer *.mp3 


Useful options 


-s signal — Sendasignal other than the default (TERM). The choices are the 
same ones listed bykill -1. 


-k seconds Ifthe program doesn’t die after the first signal, wait this many 
seconds longer and send a deadly KILL signal. 
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nice stdout --help — --version 
nice [-n level] command_line 


When invoking a system-intensive program, be nice to the 
other processes (and users) by lowering its priority. That’s what 
the nice command is for: it sets a nice level (an amount of 
“niceness”) for a process so it receives less attention from the 
Linux process scheduler.’ Here’s an example of setting a big job 
to run at nice level 7: 


> nice -n 7 sort hugefile > outfile 


Normal processes (run without nice) run at level zero, which 
you can see by running nice with no arguments: 


> nice 
0 


If you omit the -n option, the default nice level is 10. The 
superuser can also lower the nice level, increasing a process's 
priority: 


> sudo nice -n -10 some_progran 


The nice levels of processes appear in the output of ps (the NI 
column) and top (the N column). 


> ps -o pid,user,args,nice 


For programs with a lot of disk accesses, also check out the 
command ionice, which is like nice for input/output. 


1 This is called “nicing” the process. You'll hear the term used as a verb: 
“T niced the process to 12” 
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renice stdout -help — --version 
renice [-n N] [options] PID 


While the nice command invokes a program at a given nice 
level, renice changes the nice level of an already-running pro- 
cess. As a quick (though trivial) test, create a process that just 
sleeps for 2 minutes, run it in the background, and raise its nice 
level by 5: 


> sleep 120 & 

> pgrep sleep 

2673 

> renice -n 5 -p 2673 

2673 (process ID) old priority 0, new priority 5 


Ordinary users can increase the nice level of their own pro- 
cesses, while the superuser can also decrease the level (increas- 
ing the priority) and can operate on any process. The valid 
range is —20 to +20, but avoid large negative values or you 
might interfere with vital system processes. 


Useful options 


-p pid Affect the process with the given ID, pid. You can omit the -p 
and just provideaPID(renice -n 5 28734). 


-u username Affect all processes owned by the given user. 


nohup help —--version 
nohup command 


Use nohup to keep a command running after you terminate the 
shell that launched it. Ordinarily, when a shell or other process 
terminates, its child processes are sent a termination signal, 
called a hangup signal. The command nohup, which stands for 
“no hangup,’ causes the named command to ignore hangup 
signals. 
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> nohup some_long_running_command & 


If the supplied command writes to stdout or stderr, nohup redi- 
rects output to a file nohup.out in the current directory (if you 
have permission) or in your home directory (if you don't). 


flock stdin stdout --help — --version 
flock [options] lockfile command... 


Do you ever need to ensure that only one instance of a pro- 
gram runs at a time on your computer? For example, if you run 
automatic backups every hour using a command like rsync, 
there's a slight chance that a previous backup might still be 
running when the next backup launches. The flock command 
solves this sort of problem. It creates a lock file that prevents a 
command, such as a backup script, from running concurrently 
with itself. If you try to run two copies of the command at 
once with the same lock file, the second fails. For example, this 
rsync command, when run with flock and the lock file /tmp/ 
mylock, instantly fails if another instance of the same command 
is already running: 


> flock -n /tmp/mylock rsync -av diri dir2 


To see flock in action, open two shell windows and run the 
following command in each shell, one at a time (I use the sleep 
command as a demonstration, which does nothing but wait for 
a given number of seconds): 


> flock -n /tmp/mylock sleep 60 


The first command runs and the second instantly terminates 
because they have the same lockfile. This can be any file or 
directory name, which flock treats as a unique marker to 
prevent other commands from running. For example, if you 
run the preceding sleep command in one shell and a different 
command such as 1s in another, with the same lock file: 
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> flock -n /tmp/mylock ls 


then flock prevents the second command (1s) from running. 


Useful options 


-n Instantly fail if another command is already running. 
-w AN Fail after waiting N seconds, if another command is already running. 


-s Use a shared lock instead of an exclusive lock. You can run multiple 
commands simultaneously with this option, but f Lock fails if you omit 
the option. This is useful to limit the number of commands that can run 
simultaneously. 


Scheduling Jobs 


sleep Wait a set number of seconds, doing nothing. 
watch Run a command at set intervals. 
at Schedule a job for a single, future time. 
crontab Schedule jobs for many future times. 
If you need to launch programs at particular times or at regular 


intervals, Linux provides several scheduling tools with various 
degrees of complexity. 


sleep --help — --version 
sleep time_specification 


The sleep command simply waits a set amount of time. 
The given time specification can be an integer (meaning sec- 
onds) or an integer followed by the letter s (also seconds), m 
(minutes), h (hours), or d (days). For example: 


> sleep 5m Do nothing for 5 minutes 


Scheduling Jobs | 175 


sleep is useful to delay a command for a set amount of time: 


> sleep 3 && echo 'Three seconds have passed.' 
(3 seconds pass) 
Three seconds have passed. 


watch stdout -opt --help — --version 
watch [options] command 


watch executes a supplied command at regular intervals; the 
default is every two seconds. The command is passed to the 
shell (so be sure to quote or escape any special characters), 
and the results are displayed in a full-screen mode, so you can 
observe the output conveniently and see what has changed. For 
example, watch -n 1 date executes the date command once 
per second, sort of a poor man’s clock. Press *C to exit. 


Useful options 


-n seconds Set the time between executions, in seconds. 


-d Highlight differences in the output, to emphasize what has 
changed from one execution to the next. 
-g Exit when the command produces output that is different from 
the previous execution. 
at stdin stdout -- opt 


at [options] time_specification 


The at command schedules one or more shell commands to 
run later: 


> at 7am next sunday 

at> echo Remember to go shopping | mail smith 
at> Lpr SHOME/shopping-list 

at> 4D 
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<EOT> 
job 559 at 2024-09-08 21:30 


If the host is sleeping or off at the scheduled time, the job will 
run immediately when the host comes up again. The time spec- 
ifications understood by at are enormously flexible, such as: 


e A time followed by a date (not a date followed by a time) 
e Only a date (assumes the current clock time) 


e Only a time (assumes the very next occurrence, whether 
today or tomorrow) 


e A special word like now, midnight, or teatime (16:00) 


e Any of the preceding followed by an offset, like “+ 3 days” 


Dates may take many forms: december 25 2030, 25 decem 
ber 2030, december 25, 25 december, 12/25/2030, 25.12.2030, 
20301225, today, thursday, next thursday, next month, next 
year, and more. Month names can be abbreviated to three 
letters (jan, feb, mar, ...). Times are also flexible: 8pm, 8 pm, 
8:00pm, 8:00 pm, 20:00, and 2000 are equivalent. Offsets are a 
plus or minus sign followed by whitespace and an amount of 
time: + 3 days, + 2 weeks, - 1 hour, and so on.” 


If you omit part of the date or time, at copies the missing 
information from the system date and time. So next year alone 
means one year from right now, thursday alone means the 
upcoming Thursday at the current clock time, december 25 
alone means the next December 25, and 4:30pm alone means 
the very next occurrence of 4:30 p.m. in the future. 


The command you supply to at is not evaluated by the shell 
until execution time, so file patterns, variables, and other shell 
constructs are not expanded until then. Also, your current 
environment (see printenv) is preserved within each job so 


2 See the formal syntax in the file /usr/share/doc/at/timespec. 
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it executes as if you were logged in. Aliases, however, aren't 
available to at jobs, so don’t include them. 


To list your at jobs, use atq (“at queue”): 


> atq 
559 2024-09-08 21:30 a smith 


To delete a job, run atrm (“at remove”) with the job number: 


> atrm 559 


Useful options 


-f filename Read commands from the given file instead of standard 
input. 


-c job_number Print a job's commands to standard output. 


crontab stdout -file -- opt 
crontab [options] [file] 


The crontab command, like the at command, schedules jobs 
for specific times. However, crontab is for recurring jobs, such 
as “Run this command at midnight every Tuesday.” To make 
this work, you edit and save a file, called your crontab file, 
which automatically gets installed in a system directory (/var/ 
spool/cron). Once a minute, a Linux process called cron wakes 
up, checks all crontab files, and executes any jobs that are due. 


crontab -e 
Edit your crontab file in your default editor ($VISUAL). 


crontab -1 
Print your crontab file on standard output. 


crontab -r 
With no confirmation, delete your crontab file immediately 
and permanently. 
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crontab myfile 


Install the file myfile as your crontab file. 


The superuser can add the option -u username to work with 
other users’ crontab files. 


Crontab files contain one job per line. (Blank lines and com- 
ment lines beginning with “#” are ignored.) Each line has six 
fields, separated by whitespace. The first five fields specify the 
time to run the job, and the last is the job command itself. 


Minutes of the hour 

Integers between 0 and 59. This can be a single num- 
ber (30), a sequence of numbers separated by commas 
(0,15,30,45), a range (20-30), a sequence of ranges 
(@-15,50-59), or an asterisk to mean “all” You can also 
specify “every nth time” with the suffix /n; for instance, 
both */12 and 0-59/12 mean 0,12,24,36,48 (i.e., every 12 
minutes). 


Hours of the day 
Same syntax as for minutes. 


Days of the month 
Integers between 1 and 31; again, you may use sequences, 
ranges, sequences of ranges, or an asterisk. 


Months of the year 
Integers between 1 and 12; again, you may use sequences, 
ranges, sequences of ranges, or an asterisk. Additionally, 
you may use three-letter abbreviations (jan, feb, mar, ...), 
but not in ranges or sequences. 


Days of the week 
Integers between 0 (Sunday) and 6 (Saturday); again, you 
may use sequences, ranges, sequences of ranges, or an 
asterisk. Additionally, you may use three-letter abbrevia- 
tions (sun, mon, tue, ...), but not in ranges or sequences. 
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Command to execute 
Any shell command. It’s executed in your login environ- 
ment, so you can include environment variables like $HOME 
and they'll work. Use only absolute paths to your com- 
mands (e.g., /usr/bin/who instead of who) to ensure that 
cron runs the right programs, since a Linux system may 
have several programs with the same name. 


Here are some example time specifications: 
* * * * * Every minute 
45 * * * * 45 minutes after each hour (1:45, 2:45, etc.) 
45 9 * * * Every day at 9:45 a.m. 
45 9 8 * * The eighth day of every month at 9:45 a.m. 
45 9 8 12 * Every December 8 at 9:45 a.m. 
45 9 8 dec * Every December 8 at 9:45 a.m. 
45 9 * * 6 Every Saturday at 9:45 a.m. 
45 9 * * sat Every Saturday at 9:45 a.m. 
45 9 * 12 6 Every Saturday in December, at 9:45 a.m. 


45 9 8 12 6 Every Saturday in December, plus December 
8, all at 9:45 a.m. 


Here's a complete crontab entry to run a script every Saturday 
at 9:45 a.m.: 


45 9 * * sat /usr/local/bin/myscript 


If a job prints any output, cron emails a copy to the owner of 
the crontab file. 
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TIP 


Avoid long, messy shell commands in the crontab file. 
Instead, store them in shell scripts to run from crontab. 


Logins, Logouts, and Shutdowns 


systemctl Control the state of your machine and its services. 

shutdown — Shut down your local machine. 

reboot Reboot your local machine. 
Logging in and out from GNOME, KDE, or other graphical 
desktops is easy. To log out from a remote shell, just close the 
shell (run exit or logout or press “D on a line by itself). Linux 
also provides commands for rebooting and shutting down the 
computer or individual services. Never simply turn off the 


power to a Linux system: it needs a more graceful shutdown to 
preserve its filesystem. 


systemctl [options] command [arguments] 


The systemctl command controls system services. It’s part of a 
service manager called systemd. A full treatment of systemd is 
beyond the scope of this book, but I cover a few basic uses. (See 
man systemd for more details.) 


systemctl can control the system as a whole: 
sudo systemctl poweroff Shutdown the system. 


sudo systemctl reboot Reboot the system. 


sudo systemctl suspend — Suspend the system. 
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It also manages individual services, such as web servers and 
databases, with the following basic commands (among others): 


systemctl List all services and their 
statuses. 

sudo systemctl enable service_name  Makeaservice runnable, 
but do not launch it. 

sudo systemctl start service_name Launch an enabled service. 

sudo systemctl restart service_name Sameas stop followed by 
start. 

sudo systemctl reload service_name — Force arunning service to 
reread its configuration. 

sudo systemctl status service_name Print the service's current 
status. For more detailed 
status information, see 
man journalctl. 

sudo systemctl stop service_name Shut down a running 
service. 

sudo systemctl disable service_name Preventa service from 
being started. 


Service names have the suffix “service’, which you may omit. 
For example, to restart the mySQL database server, either of the 
following commands works: 


> sudo systemctl restart mysqld.service With suffix 
> sudo systemctl restart mysqld No suffix 
shutdown stdout --help 


shutdown [options] time [message] 


The shutdown command halts or reboots a Linux machine; only 
the superuser may run it. (In many Linux distros, the shutdown 
command is a symbolic link to systemctl.) Here’s a command 
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to halt the system in 10 minutes, broadcasting the message 
“scheduled maintenance’ to all users logged in: 


> sudo shutdown -h +10 "scheduled maintenance" 


The time may be a number of minutes preceded by a plus sign, 
like +10; an absolute time in hours and minutes, like 16:25; or 
the word now to mean immediately. 


With no options, shutdown puts the system into single-user 
mode, a special maintenance mode in which only root is log- 
ged in (at the system console), and all nonessential services are 
off. To exit single-user mode, either perform another shutdown 
to halt or reboot, or exit the shell with exit or “D to boot the 
system in normal, multiuser mode. 


Useful options 


-r Reboot the system. 
-h_ Halt the system. 


-k Kidding: don’t really perform a shutdown, just broadcast warning messages 
to all users as if the system were going down. 


-c Cancel a shutdown in progress (omit the time argument). 


-f Onreboot, skip the usual filesystem check performed by the fsck 
command (described in “Using Disks and Filesystems” on page 213). 


-F Onreboot, require the usual filesystem check. 


For technical information about shutdowns, single-user mode, 
and various system states, see man systemd or man init. 


reboot stdout --help 


reboot [options] 


The reboot command does exactly what it sounds like. It 
immediately reboots the computer. You might assume that 
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reboot requires root privileges, but on some distros, any user 
can run it. 


reboot has options (see the manpage) but I rarely use them. In 
some Linux distros, the reboot command is just a symbolic link 
to systemctl. 


Users and Their Environment 


logname Print your login name. 


whoami Print your current, effective username. 


id Print the user ID and group membership of a user. 
who List logged-in users, long output. 

users List logged-in users, short output. 

tty Print your terminal device name. 

last Determine when someone last logged in. 


printenv Print your environment. 


Who are you? Only the system knows for sure. This grab-bag 
of programs tells you all about users: their names, login times, 
and properties of their environment. 


logname stdout --help — --version 
logname 
The Logname command prints your login name: 


> Logname 
smith 


If this command does not work on your system, try instead: 


> echo SLOGNAME 
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whoami stdout --help — --version 


whoami 


The whoami command prints the name of the current, effective 
user. This may differ from your login name (the output of 
logname) when using the sudo command. This example distin- 
guishes whoami from Logname: 


> logname Login name 

smith 

> sudo Logname ‘The value does not change 

smith 

> whoami Effective username 

smith 

> sudo whoami ‘The value changes 

root 
id stdout --opt --help — --version 


id [options] [username] 


Every user has a unique, numeric user ID, and a default group 
with a unique, numeric group ID. The id command prints these 
values along with their associated user and group names: 


> id 
uid=500(smith) gid=500(smith) groups=500(smith) ,6(disk) 
Useful options 


-u_ Print the effective user ID and exit. 
-g Print the effective group ID and exit. 
-G_ Print the IDs of all groups to which the user belongs. 
With any of the preceding options, add -n to print user and 


group names instead of IDs, and add -r to print real IDs/names 
rather than effective ones. 


Users and Their Environment | 185 


who stdout --help — --version 
who [options] [filename] 


The who command lists all logged-in users. Users with multiple 
interactive shells appear multiple times: 


> who 

smith pts/0 Sep 6 17:09 (:0) 

barrett pts/1 Sep 6 17:10 (10.24.19.240) 
jones pts/2 Sep 8 20:58 (192.168.13.7) 
jones pts/4 Sep 3 05:11 (192.168.13.7) 


Normally, who reads its data from the file /var/run/utmp. The 
filename argument selects a different file, such as /var/log/wtmp 
for past logins or /var/log/btmp for failed logins.* 


Useful options 


-H Print a row of headings as the first line. 
--lookup For remotely logged-in users, print the hostnames of origin. 
-uU Also print each user's idle time at their terminal. 


-m Display information only about yourself, the user associated with 
the current terminal. 


-q Quick display of usernames only and a count of users. Much like the 
users command, but it adds a count. 


users stdout -help = --version 
users [filename] 


The users command prints a quick listing of users who are 
logged in. Users with multiple interactive shells appear multiple 
times: 


3 If your system is configured to log this information. 
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> users 
barrett jones smith smith smith 


Like the who command, users reads /var/log/utmp by default 
but can read from another supplied file instead. 


tty stdout help = --version 
tty 


The tty command prints the name of the terminal device 
associated with the current shell: 


> tty 
/dev/pts/4 


last stdout -opt --help — --version 
last [options] [users] [ttys] 


The last command displays a history of logins in reverse chro- 
nological order: 


> last 
bob pts/3 localhost Mon Sep 8 21:07 - 21:08 (00:01) 
sue pts/6 :0 Mon Sep 8 20:25 - 20:56 (00:31) 


bob pts/4 myhost Sun Sep 7 22:19 still logged in 


You may provide usernames or tty names to limit the output. 


Useful options 


-N Print only the latest W lines of output, where WV is a positive 
integer. 
-p time Print only users who were logged in at the given time. For 


current logins, run last -p now. 


-i Display IP addresses instead of hostnames. 
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-R Don't display hostnames. 


-xX Also display system shutdowns and changes in system runlevel 
(e.g., from single-user mode into multiuser mode). 


-f filename Read from some data file other than /var/run/wtmp; see the 
who command for more details. 


printenv stdout --opt --help — --version 
printenv [environment_variables] 


The printenv command prints all environment variables 
known to your shell and their values: 


> printenv 
HOME=/home/smith 
MAIL=/var/spool/mail/smith 
NAME=Sandy Smith 
SHELL=/bin/bash 


or only specified variables: 


> printenv HOME SHELL 
/home/smith 
/bin/bash 


User Account Management 


useradd Create an account. 
userdel Delete an account. 
usermod Modify an account. 
passwd — Change a password. 


chsh Change a user's shell. 


Linux installation software automatically creates the root 
account and usually an ordinary user account (presumably for 
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yourself). You can also create other accounts. Just remember 
that every account is a potential avenue for an intruder to enter 
your system, so give them all strong, hard-to-guess passwords. 


useradd stdout opt help 
useradd [options] username 


The useradd command lets the superuser create a user account. 
(Don't confuse it with the similarly named adduser command.) 


> sudo useradd -m smith 


Its defaults are not very useful (run useradd -D to see them), so 
be sure to supply all desired options. For example: 


> sudo useradd -d /home/smith -s /bin/bash \ 
-G games,video smith 


Useful options 

-m Create the user's home directory, and copy 
some standard files into it from your system 
skeleton directory, /etc/skel. The skeleton directory 
traditionally contains minimal (skeletal) versions 
of initialization files, like ~/.bashrc, to get new 
users started. If you prefer to copy from a different 
directory, add the -k option (-k dirname). 

-d dir Set the user's home directory to be dir. 

-s shell Set the user's login shell to be shell. 

-u uid Set the user's ID to be uid. Unless you know 
what you're doing, omit this option and accept the 
default. 

-c string Set the user's comment field (historically called the 


GECOS field). This is usually the user's full name, but 
it can be any string. 
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-g group Set the user's initial (default) group to group, which 
can either be a numeric group ID or a group name, 
and which must already exist. 


-G group1,group2,.. Make the user a member of the additional, existing 
groups group 1, group2, and so on. 


userdel stdout opt —-help 


userdel [-r] username 


The userdel command deletes an existing user. 
> sudo userdel smith 


It does not delete the user’s files (home directory, mailbox, etc.) 
unless you supply the -r option. Think carefully before deleting 
a user; consider deactivating the account instead (with usermod 
-L). And make sure you have backups of all the user’s files 
before deleting them—you might need them someday. 


usermod stdout --opt --help 
usermod [options] username 


The usermod command modifies the given user’s account in 
various ways, such as changing a home directory: 


> sudo usermod -d /home/another smith 


Useful options 


-a When adding the user to a group (-G), preserve the 
user's existing group memberships. 


-c string Set the user's comment field (historically called the 
GECOS field). This is usually the user's full name, but 
it can be any string. 
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-ddir 


-L username 


-s shell 


-g group 


-G group1,group2,... 


passwd 


Change the user's home directory to dir. 


Change the user's login name to username. Think 
carefully before doing this, in case anything on your 
system depends on the original name. And don’t 
change system accounts (root, daemon, etc.) unless 
you really know what you're doing! 


Change the user's login shell to shell. 


Change the user's initial (default) group to group, 
which can either be a numeric group ID or a group 
name, and which must already exist. 


Make the user a member only of the existing groups 
group 1, group2, and so on. WARNING: if the user 
currently belongs to other groups and you don’t list 
them here, usermod removes the user from the 
other groups. To prevent this behavior and preserve 
the user's existing groups, add the - a option. 


Disable (lock) the account so the user cannot log in. 


Unlock the account after a lock (-L) operation. 


stdout --opt --help 


passwd [options] [username] 


The passwd command changes a login password, yours by 


default: 


> passwd 


or another user’s password if run by the superuser: 


> sudo passwd smith 


passwd does have options, most of them related to password 
expiration. Use them only in the context of a well thought-out 


security policy. 
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chsh stdout --opt --help — --version 
chsh [options] [username] 


The chsh (change shell) command sets your login shell pro- 
gram. Invoked without a username, chsh affects your account; 
invoked with a username (by root), it affects that user. With no 
options, chsh prompts you for the desired information: 


> chsh 
Password: XXXXXXXX 
New shell [/bin/bash]: /bin/tcsh 


The new shell must be listed in the file /etc/shells. 


Useful options 


-s shell Specify the new shell. 


-1 List all permissible shells installed on your system. 


Group Management 


groups Print the group membership of a user. 
groupadd Create a group. 

newgrp  Useanew group membership immediately. 
groupdel Delete a group. 

groupmod Modify a group. 


A group is a set of accounts treated as a single entity. If you give 
permission for a group to take some action (such as modify a 
file), then all members of that group can take it. For example, 
give full permissions for the group “friends” to read, write, and 
execute the file /tmp/sample: 


> groups 
users smith friends 

> chgrp friends /tmp/sample 
> chmod 770 /tmp/sample 
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> 1s -l /tmp/sample 
-rwxrwx--- 1 smith friends 2874 ... /tmp/sample 


To add users to a group, run usermod -aG or edit /etc/group as 
root. To change the group ownership of a file, recall the chgrp 
command from “Properties of Files” on page 83. 


groups stdout --help —--version 


groups [usernames] 


The groups command prints the Linux groups to which you 
belong, or to which other users belong: 


> whoami 

smith 

> groups 

smith users 

> groups jones root 

jones : jones users 

root : root bin daemon sys adm disk wheel src 


groupadd opt --help 
groupadd [options] group 


The groupadd command creates a group. (Don't confuse it with 
the similarly named addgroup command.) In most cases, add 
the -f option to prevent duplicate groups from being created: 


> sudo groupadd -f friends 
Useful options 


-g gid Specify a numeric group ID, gid. Normally, groupadd chooses it. 


-f If the named group exists already, complain and exit. 
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hewgrp ~help 
newgrp [-] [group] 


When you're added to a new group (e.g., with usermod -aG), 
normally the change isn't effective until your next login. The 
newgrp command avoids this hassle. Run newgrp with the new 
group name as an argument, and it launches a fresh shell with 
your current group ID set to that group. In this way, you can 
immediately use newly granted group privileges. The effect 
lasts only while the new shell runs, but it beats logging out and 
back in. Exit the shell to restore your default group ID. 


For example, suppose you've just been added to the group 
video. If you haven't logged out yet, you won't see video among 
your groups. 


> groups View your active groups 
smith sudo docker 


Run newgrp to set your group ID to video: 


> newgrp video Launch a shell with group ID = video 
> groups Now “video” is your default group 
video sudo docker smith 

> exit Set group ID back to default 


The only option, a single dash, causes newgrp to set up your 
environment as if youd just logged in, like su -1 does. 


groupdel opt ~help 
groupdel group 
The groupdel command deletes an existing group: 


> sudo groupdel friends 


Before removing a group, identify all files owned by that group 
so you can clean them up later: 
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> sudo find / -group friends -print > /tmp/friend. files 


because groupdel does not change the group ownership of any 
files. It simply removes the group name from /etc/group. Any 
files owned by the deleted group will retain the now-obsolete 
group ID. 


groupmod stdout fle opt --help 
groupmod [options] group 


The groupmod command modifies the given group, changing its 
name or group ID: 


> sudo groupmod -n newname friends 
groupmod does not affect any files owned by this group: it sim- 


ply changes the ID or name in the system's records. 


Useful options 


-n name Change the group’s name to name (safe). 


-g gid Change the group’s ID to gid (risky). Any files with the original group 
ID will now have invalid group ownership and need cleanup. 


Installing Software Packages 


dnf Standard package manager for RPM files (CentOS, Fedora, Red Hat, 
Rocky, etc.). 

yum Older package manager for RPM files. 

rpm Manipulate RPM packages locally. 

apt Standard package manager for DEB files (Debian, Deepin, 


elementary 0S, Kodachi, Linux Lite, MX, Mint, Nitrux, POP!_0S, 
Rescatux, Ubuntu, Zorin 0S, etc.). 


aptitude Alternative package manager for DEB files. 


dpkg Manipulate DEB packages locally. 
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emerge _ Portage package manager for Gentoo Linux. 


pacman Package manager for Arch Linux (plus Garuda, Endeavour0S, 
Manjaro, etc.). 


zypper Package manager for openSUSE Linux. 
flatpak — Container-based package manager. 


snap Container-based package manager. 


Your Linux distro comes with a package manager application 
to install software packages via the command line, GUI tools, 
or both. Confusingly, every package manager has unique com- 
mands and may use a different file format for packages. So, one 
of your first tasks as a Linux user is to learn which package 
manager is standard for your distro. If you aren't sure which 
Linux distro you're running, one of the following commands 
should give you a clue: 


> cat /etc/issue 

Ubuntu 22.04.2 LTS \n \l Running Ubuntu Linux 
> more /etc/*-release 

DISTRIB_ID=Ubuntu 

DISTRIB_RELEASE=22.04 

DISTRIB_CODENAME=jammy 


Then search for your distro’s package manager on the web, or 
just run each package management command in my list to see 
which ones are installed. The most common package types and 
managers are: 


Debian, dpkg, or .deb packages 
Used by Debian, Ubuntu, and other distros. I cover the 
package management commands apt, aptitude, and dpkg. 


RPM packages 
Used by Red Hat, Fedora, CentOS, and other distros. I 
cover dnf, yum, rpm, and zypper. 


These package types and others (some of which I cover) install 
software into system directories like /usr/bin. Another kind 
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of package manager takes a different approach and runs its 
software in a restricted mini-environment called a container. 
Examples are Snap and Flatpak. 


stdout --version 


dnf 


dnf [options] [packages] 


--help 


dnf is the latest package manager for RPM packages (.rpm 


files). The following table lists common operations: 


Action dnf command 


Search for a package that meets your 
needs (supports wildcards * and ?). 


Check if a package is installed. 
Download but don’t install a package. 
Download and install a package. 
Install a package file. 

Learn about a package. 

List the contents of a package. 


Discover which package an installed file 
belongs to. 


Update an installed package. 
Remove an installed package. 


List all packages installed on the system 
(tip: pipe through Less). 


Check for updates for all installed 
packages. 


Update all packages on the system. 


Update the distro to the next version. 


dnf search string 


dnf list installed package 
dnf download package 

sudo dnf install package 
sudo dnf install file.rpm 
dnf info package 

rpm -ql package 

dnf provides /path/to/file 


sudo dnf upgrade package 
sudo dnf remove package 


dnf list installed 


dnf check-update 


sudo dnf upgrade 


sudo dnf system-upgrade 
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yum stdout -help — --version 
yum [options] [packages] 


yum is an older package manager for RPM packages (.rpm files), 
which has been largely replaced by dnf. The following table 
lists common operations with yum. For operations on local files, 
which yum does not provide, use the rpm command directly. 


Action yum command 


Search for a package that meets yum search string 
your needs (supports wildcards * 


and ?). 

Check if a package is installed. yum list installed package 
Download a package but don’t sudo yum --downloadonly 
install it.? install package 


Download and install a package. © sudo yum install package 


Install a package file. rpm -ivh file.rpm 

Learn about a package. yum info package 

List the contents of a package. rpm -ql package 

Discover which package an yum provides /path/to/file 
installed file belongs to. 

Update an installed package. sudo yum update package 
Remove an installed package. sudo yum remove package 


List all packages installedonthe = yum list installed 
system (tip: pipe through Less). 


Check for updates for allinstalled © yum check-update 
packages. 


Update all packages on the sudo yum update 
system. 


@ May require the downloadonLy plug-in. To install it, run sudo yum 
install yum-downloadonly 
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rpm stdout --help = --version 
rpm [options] [files] 


If you prefer to download and install RPM packages by hand, 
use the rpm command. Unlike dnf or yum, rpm works locally 
on your computer: it does not search software archives on the 
internet for new packages. 


RPM filenames typically take the following form <name>- 
<version>.<architecture>.rpm. For example, the filename 
emacs-29. 1-2.x86_64.rpm indicates the emacs package, version 
29.1-2, for 64-bit x86 processors. Be aware that rpm sometimes 
requires a filename argument (like emacs-29.1-2.x86_64.rpm) 
and other times just the package name (like emacs). The fol- 
lowing table lists common operations: 


Check if a package is installed. rpm -q package 

Install a package file. sudo rpm -ivh file.rpm 

Learn about a package. rpm -qi package 

List the contents of a package. rpm -ql package 

Discover which package an rpm -qf /path/to/file 

installed file belongs to. 

Update an installed package. sudo rpm -Uvh package_file.rpm 
Remove an installed package. sudo rpm -e package 


List all packages installed onthe rpm -qa 
system (tip: pipe through Less). 
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APT stdout -help — --version 


apt subcommand [options] packages 
dpkg [options] packages 
The APT (Advanced Packaging Tool) suite of commands can 


install, remove, and manipulate Debian Linux (.deb) packages. 
The following table lists common operations: 


Action APT command 


Retrieve the latest information about sudo apt update 
available packages before running 
other commands. 


Search for a package that meets your. apt search string 
needs. 


Check if a package is installed. apt policy package 


Download but don’t install apackage. sudo apt install -d package 


Download and install a package. sudo apt install package 
Install a package file. sudo apt install file.deb 
Learn about a package. apt show package 
List the contents of a package. dpkg -L package 


Discover which package an installed dpkg -S /path/to/file 
file belongs to. 


Update an installed package. sudo apt upgrade package 
Remove an installed package. sudo apt remove package 
Remove an installed package and sudo apt purge package 
associated files. 

List all packages installed on the apt list --installed 

system (tip: pipe through Less). 

Check for updates for all installed sudo apt list --upgradable 
packages (runsudo apt update 

first). 
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Action APT command 


Update all packages on the system. sudo apt upgrade 


Update the distro to the next version. sudo apt dist-upgrade 


aptitude stdout --help — --version 
aptitude [options] [packages] 


aptitude is another package manager for the command line 
that manipulates Debian (.deb) packages. When run with no 
arguments, it provides a full-terminal interactive interface to 
the APT system: 


> sudo aptitude 


aptitude can also run some, but not all, APT operations from 
the command line. The following table lists common opera- 
tions, including a few that aptitude does not support and the 
appropriate apt or dpkg commands to run instead. 


Action aptitude command 


Retrieve the latest sudo aptitude update 
information about available 

packages before running 

other commands. 


Search for a package that aptitude search string 
meets your needs. 


Check if a package is aptitude show package 
installed (see “State” in the 
output). 


Download but don’tinstalla aptitude download package 
package. 


Download and install a sudo aptitude install package 
package. 


Installing Software Packages | 201 


Install a package file. sudo apt install file.deb 
Learn about a package. aptitude show package 

List the contents of a dpkg -L package 

package. 


Discover which packagean dpkg -S /path/to/file 
installed file belongs to. 


Update an installed sudo aptitude safe-upgrade package 
package. 

Remove an installed sudo aptitude remove package 
package. 


List all packages installed aptitude search ~i 
on the system (tip: pipe 
through less). 


Check for updates for all aptitude --simulate full-upgrade 
installed packages. 


Update all packages onthe sudo aptitude full-upgrade 
system. 


emerge stdout help —--version 


emerge [options] [arguments] 

emaint [options] subcommand 

equery [options] subcommand [arguments] 

The emerge command controls the package manager in Gentoo 


Linux, called Portage. Before working with Portage packages 
for the first time, run the following command: 


> sudo emerge gentoolkit = Install additional Portage commands 
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Action emerge command 


Retrieve the latest sudo emaint -a sync 
information about available 

packages before running 

other commands. 


Search for a package that emerge -s string 
meets your needs, by name. 


Search for a package that emerge -S string 
meets your needs, by 
description. 


Check if a package is equery list "*" | grep package 
installed. 


Download but don’tinstalla sudo emerge -f package 
package. 


Download and install a sudo emerge package 

package. 

Learn about a package. sudo equery meta [--description] 
package 

List the contents of a equery files package 

package. 


Discover which package an equery belongs /path/to/file 
installed file belongs to. 


Update an installed package. sudo emerge -u package 


Remove an installed sudo emerge -cav package 
package. 


List all packages installedon equery list "*" 
the system (tip: pipe through 
less). 


Check for updates for all emerge -puD world 
installed packages. 


Update all packages onthe = sudo emerge -uD world 
system. 
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If emerge wont remove a package because of dependencies, try 
including the dependencies on the command line: 


> sudo emerge -cav my/package fails 
Calculating dependencies... done! 
my/package-29.3 pulled in by: 
other/pack-16.1 requires ... dependency! 


>>> No packages selected for removal by depclean 
> sudo emerge -cav my/package other/pack = succeeds 
Would you Like to unmerge these packages? [Yes/No] Yes 


pacman stdout help —--version 
pacman subcommand [options] [arguments] 


The pacman command is a package manager for Arch Linux. 
Arch packages are typically tar files compressed with the com- 
mand zstd. The following table lists common operations: 


Action pacman command 


Retrieve the latest information © sudo pacman -Sy 
about available packages before 
running other commands. 


Search for a package that pacman -Ss string 
meets your needs (by regular 
expression). 


Check if a package is installed. pacman -Q package 


Download but don’t install a sudo pacman -Sw package 

package. 

Download and install a package. sudo pacman -S package 

Install a package file. sudo pacman -U file.pkg.tar.zst 
Learn about a package. pacman -Qi package 


List the contents of a package. pacman -Ql package 


204 | Chapter 3: System Administration Basics 


Action 


Discover which package an 
installed file belongs to. 


Update an installed package. 
Remove an installed package. 


List all packages installed on 
the system (tip: pipe through 
less). 


Check for updates for all 
installed packages. 


Update all packages on the 
system. 


pacman command 


pacman -Qo /path/to/file 


sudo pacman -S package 
sudo pacman -R package 
Pacman -Qe 


sudo pacman -Syup 


sudo pacman -Syu 


zypper 


stdout help — --version 


zypper [options] subcommand [subcommand_opts] [arguments] 


The zypper command is a package manager for openSUSE 
Linux. It uses RPM packages under the hood. The following 
table lists common operations: 


Action zypper command 


Retrieve the latest information 
about available packages before 
running other commands. 


Search for a package that meets 
your needs. 


Check if a package is installed. 


Download but don’t install a 
package. 


Download and install a package. 


Install a package file. 


sudo zypper refresh 


zypper search string 


zypper search -i package 


sudo zypper install -d package 


sudo zypper install package 


sudo rpm -ivh file.rpm 
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Learn about a package. zypper info package 

List the contents of a package. rpm -ql package 

Discover which package an rpm -qf /path/to/file 
installed file belongs to. 

Update an installed package. sudo zypper update package 
Remove an installed package. sudo zypper remove package 


List all packages installed onthe © zypper search -i 
system (tip: pipe through Less). 


Check for updates for allinstalled © sudo zypper update --dry-run 
packages. 


Update all packages on the sudo zypper update 
system. 
Update the distro to the next sudo zypper dist-upgrade 
version. 
flatpak stdout --help — --version 


flatpak subcommand [options] [arguments] 


Flatpak is a system for installing software packages, called 
“Flatpaks,” that run in a restricted environment, called a con- 
tainer or sandbox. A container includes all the package's depen- 
dencies. Use the flatpak command to install, update, and 
remove packages. You may need to add a remote repository 
first: 


> sudo flatpak remote-add --if-not-exists flathub \ 
https: //flathub.org/repo/flathub. flatpakrepo 


Flatpaks have limited access to the host filesystem, but for 
the most part, they operate like ordinary applications...except 
when you run them at the command line. Flatpaks must be run 
with the flatpak command. So if you've installed GNU Emacs, 
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for example, find out its Flatpak ID, which is a three-part string 
containing dots, such as org.gnu.emacs. Then run the program 
by its ID: 

> flatpak List | grep emacs 

ID sieve 

org.gnu.emacs ... The flatpak ID 

> flatpak run org.gnu.emacs 


If typing the ID is annoying, define an alias: 
alias emacs='flatpak run org.gnu.emacs' 


The following table lists common operations for working with 
Flatpaks system-wide, which often requires superuser privi- 
leges. To install Flatpaks just for yourself, run flatpak --user 
instead of sudo flatpak. 


Action Flatpak command 


Add a remote repository for sudo flatpak remote-add 
downloading Flatpaks. --if-not-exists name url 


List the remotes installed flatpak remotes 


on your system. 

Search fora package that flatpak search string 

meets your needs. 

Check if a package is flatpak list | grep package 
installed. 

Download and install a sudo flatpak install package 
package. 

Install a package file. sudo flatpak install /path/to/file 
Run a package. flatpak run package_id 

Learn about a package. flatpak info package_id 

Update an installed sudo flatpak update package_id 
package. 

Remove an installed sudo flatpak uninstall package_id 
package. 
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Action Flatpak command 


List all packages installed  flatpak list 
on the system (tip: pipe 


through less). 

Update all packages onthe sudo flatpak update 

system. 

snap stdout -help —--version 


snap [options] subcommand [subcommand_options] 


Snap is a system for installing software packages, called “snaps,” 
that run in a restricted environment, called a container or sand- 
box. A container includes all the package's dependencies. Snaps 
have limited access to the host filesystem, but for the most part, 
they operate like ordinary applications. Use the snap command 
to install, update, and remove packages. The following table 
lists common operations: 


snap command 


Search for a package that meets your snap find string 
needs. 


Check if a package is installed. snap list | grep package 


Download but don’t install a package. sudo snap download package 


Download and install a package. sudo snap install package 
Learn about a package. snap info package 

List the contents of a package. ls /snap/package/current 
Update an installed package. sudo snap refresh package 
Remove an installed package. sudo snap remove package 


List all packages installed on the system snap list 
(tip: pipe through Less). 
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Action snap command 


Check for updates for all installed snap refresh --list 
packages. 
Update all packages on the system. sudo snap refresh 


Installing Software from Source Code 


configure Prepare to build software manually with make. 


make Build software from source code. 


Package managers, described in “Installing Software Packages” 
on page 195, generally require superuser privileges to install 
software. As an ordinary user, you can also install software 
within your home directory. The process requires more steps 
and more understanding than using a package manager. 


Downloading the Source Code 


Plenty of Linux software is distributed as source code to be 
downloaded and built on your local system. The two most 
common forms of distribution are: 


e Compressed TAR or ZIP files found on websites 


¢ Git repositories, cloned from servers like GitHub 


I briefly explain each and then show how to build the source 
code with the commands configure and make. 


Method 1: Download and unpack a TAR or ZIP file 


A compressed TAR file, or “tarball,” is just a collection of files 
packed up by tar. The file is usually compressed with gzip 
and has the filename extension .tar.gz or .tgz, or bzip2 with the 
extension .tar.bz2 or .tbz. Similarly, a .zip file is a collection of 
files packed by zip. To unpack files: 
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1. List the package contents. Assure yourself that each file, 
when extracted, won't overwrite something precious on 
your system, either accidentally or maliciously:* 


> tar -tvf package.tar.gz | less gzip 
> tar -tvf package.tar.bz2 | less bzip2 
> unzip -l package.zip | less zip 


2. If satisfied, extract the files into a new directory: 
> mkdir newdir 
> tar -xvf package.tar.gz -C newdir gzip 
> tar -xvf package.tar.bz2 -C newdir bzip2 
> unzip -d newdir package.zip | less zip 


If this worked, continue to “Building and Installing the Code” 
on page 210. 


Method 2: Clone a Git repository 


To obtain software from GitHub or a similar repository, copy 
the URL of the desired repository and pass it to the git clone 
command, which should look something like this: 


> git clone git@github.com:username/repository.git 


git clone downloads a copy of the repository to your local 
system. Now you're ready to build the software. 


Building and Installing the Code 


Once you've downloaded and/or extracted the source code, 
build the software. Briefly, the usual sequence of commands is: 


> ./configure PREFIX=any_writable_directory 
> make 
> make install 


4 A malicious archive could include an absolute file path like /etc/ 
passwd which, if extracted, could overwrite your system password file. 
Nasty. 
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In more detail, the sequence is as follows: 


1. In the source code directory, read the extracted file named 
INSTALL or README. For example: 
> less INSTALL 


2. Usually, you're instructed to run a provided script called 
configure, then run make, then run make install. To view 
the options for the configure script, run this command 
(the output is usually long): 

> ./configure --help | less 


The most important option for our purposes is --prefix, 
which sets the installation directory. For example, to 
install the software in your home directory, in a subdirec- 
tory named packages, youd run: 

> ./configure --prefix=SHOME/packages 


If you omit the --prefix option, then configure arranges 
to install the software system-wide, and you'll need super- 
user privileges later to complete the installation. 

> ./configure 


If configure fails, it usually means your local system is 
missing some prerequisite software. Read the output of 
configure carefully, install what is missing, and try again. 


3. Once configure succeeds, build (compile) the software by 
running make. It performs all necessary steps to prepare 
the software for installation, without installing it. 

> make 


If make fails, read the error output carefully, search the 
web for solutions, and if appropriate, file a bug with the 
maintainers of the software you're building. 


4. If make completed successfully, and you ran configure 
with the --prefix option, complete the installation: 
> make install 
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Or, if you omitted --prefix to install the software system- 
wide, use sudo: 
> sudo make install 


If you originally ran configure with the --prefix option, the 
installed software will be in the directory you specified, usually 
in a subdirectory named bin. Optionally, add this bin directory 
to your search path (see “Search Path” on page 41). If you 
installed the software in $HOME/packages as in my example, 
append its bin subdirectory to your PATH with this command: 


> PATH=SPATH: SHOME/packages/bin 


Add this line to your shell configuration file (see “Tailoring 
Shell Behavior” on page 58) to make the new software available 
to your future shells. 
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CHAPTER 4 
Filesystem Maintenance 


Using Disks and Filesystems 


df Display available space on mounted filesystems. 
Lsblk List disks and other block devices. 
mount Make a disk partition accessible. 
umount Unmounta disk partition (make it inaccessible). 
fsck — Checka disk partition for errors. 
Linux systems can have multiple disks or partitions. In casual 


conversation, these are variously called devices, filesystems, 
volumes, even directories. I'll try to be more precise. 


A disk is a mass storage device, which may be divided into 
partitions that act as independent devices. Disks and partitions 
are represented on Linux systems as special files in the direc- 
tory /dev. For example, /dev/sda7 could be a partition on your 
hard drive. Some common devices in /dev are: 


sda First block device, such as SCSI, SATA, or USB hard drives; partitions are 
sdal,sda2,.... 

sdb Second block device; partitions are sdb7, sdb2, .. .. Likewise for sdc, 
sdd, etc. 
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md0 First RAID device; partitions are md0p1, mdOp2, .... Likewise for md17, 
md2, etc. 


nvme0n1 First NVMe SSD device; partitions are nvmeOn7p1, nvmeOnip2, .... 
Likewise for nvmein7, nvme2n7, etc. The second integer, like the 1 in 
nvmeOn1p2, is called the namespace ID, and most users can ignore it. 


Before a partition can hold files, it is formatted by a program 
that creates a filesystem on it (see “Creating and Modifying 
Filesystems” on page 219). A filesystem defines how files are 
represented; examples are ext4 (a Linux journaling filesystem) 
and NTFS (a Microsoft Windows filesystem). Formatting is 
generally done for you when you install Linux. 


After creating a filesystem, make it available by mounting its 
partition on an empty directory.' For example, if you mount a 
Windows filesystem on a directory /mnt/win, it becomes part 
of your system's directory tree, and you can create and edit files 
like /mnt/win/myfile.txt. Mounting generally happens automat- 
ically at boot time. You can also unmount partitions to make 
them inaccessible via the filesystem for maintenance. 


df stdout -opt --help  --version 
df [options] [disk devices | files | directories] 


The df (disk free) command shows you the size, used space, 
and free space on a given disk partition. If you supply a file 
or directory, df describes the disk device on which that file or 
directory resides. With no arguments, df reports on all moun- 
ted filesystems: 


1 You can mounta filesystem on a nonempty directory, but the direc- 
tory’s contents become inaccessible until you unmount. 
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> df 

Filesystem 1k-blocks Used Avail Use% Mounted on 
/dev/sda 1011928 225464 735060 24% / 
/dev/sda9 521748 249148 246096 51% /var 
/dev/sda8 8064272 4088636 3565984 54% /usr 
/dev/sda10 8064272 4586576 3068044 60% /home 


The df command may list all sorts of devices besides disks. To 
limit the display to disks, try these options (and create an alias 
if it’s helpful): 


> df -h -x tmpfs -x devtmpfs -x squashfs 


Useful options 
-k List sizes in KB (the default). 
-m List sizes in MB. 


-BN Display sizes in blocks of N bytes. (Default = 1024) 


-h Print human-readable output and choose the most appropriate unit 

-H for each size. For example, if your two disks have 1 gigabyte and 25 
KB free, respectively, df -h prints 1G and 25K. The -h option uses 
powers of 1024, whereas -H uses powers of 1000. 


-1 Display only local filesystems, not networked filesystems. 

-T Include the filesystem type (ext3, vfat, etc.) in the output. 
-t type Display only filesystems of the given type. 

-x type Don't display filesystems of the given type. 


-t Inode mode. Display total, used, and free inodes for each filesystem, 
instead of disk blocks. When all inodes on a filesystem are used, the 
filesystem is “full” even if free disk space remains. 
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Isblk stdout --opt --help — --version 
lsblk [options] [devices] 


The lsblk command lists the mass storage devices, known as 
block devices, available on a Linux system, such as hard disks, 
SSDs, and RAM disks. 


> Usblk 

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS 
sda 8:0 0 20G © disk 

/sda1 8:1 0 1M © part 

/sda2 8:2 © 513M © part /boot/efi 
/sda3 8:3 @ 19.5G 0 part / 

sdb 8:80 0 7.6G © disk 

tsdb1 8:81 1 7.6G © part /mnt/usb-key 


The output shows a hard drive at /dev/sda with three partitions, 
and a USB thumb drive at /dev/sdb with a single partition. 
lsblk has a ton of formatting options and can limit itself to 
particular devices. 


> Usblk -o NAME,SIZE /dev/sda 
NAME SIZE 
sda 20G 
/sda1 1M 
-sda2 513M 
L-sda3 19.5G 


Useful options 


-1 Display a simple list instead of a tree. 
-a Show all block devices, including those normally hidden. 
-f Add information about the filesystems on each device. 


-o columns Print only the given columns, which you provide as a comma- 
separated list. View the available columns with LsbLk 
--help. 


-J Print the list in JSON format for easy processing by programs. 
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mount stdout -opt --help — --version 
mount [options] [device | directory] 


The mount command makes a partition accessible. Most com- 
monly it handles disk drives (say, /dev/sdal) and removable 
media (e.g., USB keys), making them accessible via an existing 
directory (say, /mnt/mydir): 


v 


sudo mkdir /mnt/mydir 

ls /mnt/mydir Notice it’s empty 
> sudo mount /dev/sdai /mnt/mydir 

> 1s /mnt/mydir 

file1l file2 file3 Files on the mounted partition 
> df /mnt/mydir 

Filesystem 1K-blocks Used Avail Use% Mounted on 
/dev/sda1 1011928 285744 674780 30% /mnt/mydir 


v 


mount has many uses; I discuss only the most basic. In most 
common cases, mount reads the file /etc/fstab (filesystem table, 
pronounced “F S tab”) to learn how to mount a desired disk. 
For example, if you run mount /usr, the mount command looks 
up “/usr” in /etc/fstab, whose line might look like this: 


/dev/sda8 /usr ext4 defaults 1 2 


Here mount learns that device /dev/sda8 should be mounted 
on /usr as a Linux ext4-formatted filesystem with default 
options. Mount it with either of these commands: 


> sudo mount /dev/sda8 By device 
> sudo mount /usr By directory 


mount is run typically by the superuser, but common removable 
devices like USB keys and DVDs often can be mounted and 
unmounted by any user. 


Useful options 


-t type Specify the type of filesystem, such as ext4 orntfs. 


-1 List all mounted filesystems; works with - t too. 
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-a Mount all filesystems listed in /etc/fstab. Ignores entries that include 
the noauto option. Works well with - t too. 


-r Mount the filesystem read-only (see the manpage for disclaimers). 


umount stdout -opt --help — --version 
umount [options] [device | directory] 


umount does the opposite of mount: it makes a disk partition 
unavailable via the filesystem.” For instance, if you've mounted 
a USB thumb drive, umount it before you unplug it: 


> umount "/media/smith/My Vacation Photos" 


Always unmount any removable medium before ejecting it, 
particularly if it’s writable, or you risk damage to its filesystem. 
To unmount all mounted devices: 


> sudo umount -a 


Don't unmount a filesystem that’s in use; in fact, the umount 
command refuses to do so for safety reasons. 


fsck stdout --opt --help — --version 
fsck [options] [devices] 


The fsck (filesystem check) command validates a Linux disk 
filesystem and, if requested, repairs errors found on it. fsck 
runs automatically when your system boots, or manually. In 
general, unmount a device before checking it, so no other 
programs are operating on it at the same time: 


2 Notice the spelling is “umount,” not “unmount? 
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> sudo umount /dev/sda10 

> sudo fsck -f /dev/sdai0 

Pass 1: Checking inodes, blocks, and sizes 

Pass 2: Checking directory structure 

Pass 3: Checking directory connectivity 

Pass 4: Checking reference counts 

Pass 5: Checking group summary information 

/home: 172/1281696 files (11.6% non-contiguous), ... 


You cannot use fsck to fix your root filesystem while your 
system is running normally. Boot first on a Linux USB thumb 


drive or other rescue media, then run fsck. 


fsck is a frontend for a set of filesystem-checking commands 
found in /sbin, with names beginning “fsck”. Only certain types 


of filesystems are supported; list them with the command: 


> 1s /sbin/fsck.* | cut -d. -f2 | column 
cramfs ext3 fat hfsplus msdos 
ext2 ext4 hfs minix vfat 


Useful options 


-A_ Check all disks listed in /etc/fstab, in order. 
-f Force fsck to run even if no errors are apparent. 


-N_ Print a description of the checking that would be done, but exit without 
performing any checking. 


-r_ Fix errors interactively, prompting before each fix. 


-a Fix errors automatically (use only if you really know what you're doing; if 
not, you can seriously mess up a filesystem). 


Creating and Modifying Filesystems 


mkfs Format (create a filesystem on) a disk partition. 
resize2fs Groworshrink a disk partition. 


e2label —_ Change the volume label on a disk partition. 
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Disk-related operations like partitioning and formatting can be 
complex at the command line. In general, for anything more 
complicated than formatting a single partition, I recommend 
using a graphical application such as gparted. Honestly, it’s 
easier and less error-prone. 


Nevertheless, I still run a few operations at the command line 
that are quick and easy. One is listing the partitions of a storage 
device like /dev/sda with fdisk: 


> sudo fdisk -l /dev/sda 
Disk /dev/sda: 20 GiB, 21474836480 bytes, 


Device Start End Sectors Size Type 


/dev/sda1 2048 4095 2048 1M BIOS boot 
/dev/sda2 4096 1054719 1050624 513M EFI System 


/dev/sda3 1054720 41940991 40886272 19.5G Linux 
or similarly with parted: 
> sudo parted /dev/sda -- print 


Another is exporting the partition table of a storage device for 
safekeeping. (Store it on a USB thumb drive or other device, 
not the disk you're working on!) 


> sudo sfdisk -d /dev/sda > /mnt/thumb/sda.txt 


Later, if you mess up a risky partitioning operation, you can 
restore the partition table (but be careful to specify the correct 
disk device or you'll overwrite the wrong partition table): 


> sudo sfdisk /dev/device < /mnt/thumb/sda.txt 


The commands that follow are also relatively basic operations 
on disks and filesystems without graphical tools. 
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mkfs stdout --help — --version 


mke2fs [options] device 

mkfs.ext3 [options] device 

mkfs.ext4 [options] device 

mkntfs [options] device 

mkfs.ntfs [options] device ...and many other variations... 

The mkfs family of commands formats a Linux storage device 


for a variety of filesystems. The storage device is usually a 
partition, such as /dev/sdbl1. 


WARNING 


mkfs erases a storage device. Make sure the device name 
you provide is the correct one! 


Examples: 


> sudo mkfs.ext4 /dev/device Standard Linux filesystem 


> sudo mke2fs /dev/device Standard Linux filesystem 
> sudo mkfs.ntfs /dev/device = Microsoft Windows filesystem 
> sudo mkntfs /dev/device Microsoft Windows filesystem 


As you can see, most of the command names are “mkfs” fol- 
lowed by a dot and a filesystem type, like mkfs.ext4. They 
may also have alternate names (links) with the filesystem type 
embedded in the middle of “mkfs’, such as mke2fs for an “ext” 
filesystem. To list all such commands installed on your system, 
run: 


> 1s /usr/*bin/mkfs.* 

/usr/sbin/mkfs.ext2 /usr/sbin/mkfs.hfs 
/usr/sbin/mkfs.ext3 /usr/sbin/mkfs.minix 
/usr/sbin/mkfs.ext4 /usr/sbin/mkfs.msdos 
/usr/sbin/mkfs. fat /usr/sbin/mkfs.ntfs 
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Useful options 


-n Dry-run mode: don’t format anything. Just display what would be 
done. 


-L name Label the formatted volume with the given name, which can be up to 
16 bytes long. 


-b N Set the block size to N bytes. 


resize2fs 
resize2fs [options] device [size] 
The resize2fs command grows or shrinks a standard Linux 
filesystem of type ext2, ext3, or ext4. To enlarge a filesystem: 
1. Confirm that the device has enough free space immedi- 
ately following the current partition. 
2. Unmount the filesystem. 


3. Enlarge its disk partition with gparted or similar program. 
(This requires free space just after the current partition.) 


4. Check the filesystem with fsck. 


5. Run resize2fs with appropriate arguments. In modern 
kernels, the filesystem may be mounted during resizing. 


To shrink a filesystem: 


1. Confirm with df that the data in the filesystem (the “Used” 
column) will fit within the proposed new size. 


2. Unmount the filesystem. 


3. Run resize2fs with appropriate arguments. 
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4. Shrink its disk partition with gparted or a similar 
program. 


5. Check the filesystem with fsck. 


To resize a filesystem on /dev/sdal, assuming you've already 
completed any checking and partitioning, run resize2fs either 
with or without a size: 


> sudo resize2fs /dev/sda1 100G = Resize to 100 GB 
> sudo resize2fs /dev/sda1 Resize to the partition size 


Sizes can be an absolute number of blocks, like 12345690, or 
a size followed by K (KB), M (MB), G (GB), T (terabytes), or s 
(512-byte sectors). The values are powers of two, so 1K means 
1024, not 1000, and so on. 


If you resize filesystems often, make your life easier with logical 
volume management (LVM), as explained in “Logical Volumes 
for Flexible Storage” on page 229, or a more modern filesystem, 
as in “ZFS: A Modern, Do-It-All Filesystem” on page 235. 


Useful options 


-f Force the resizing operation, even if resize2fs complains. 


-p Display the progress of the operation as it runs. 


e2label 


e2label device [label] 


A label is a nickname for a filesystem. The e2label command 
sets or prints the label of a standard Linux filesystem of type 
ext2, ext3, or ext4. Filesystems don't require labels, but they're 
convenient for referring to filesystems in /etc/fstab. 


> sudo e2label /dev/sdb1 backups Assign a label 
> sudo e2label /dev/sdbi Print a label 
backups 
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RAID Arrays for Redundancy 


mdadm Manage RAID arrays. 


RAID (Redundant Array of Independent Disks) is a technique 
that distributes a computer’s data across multiple disks, trans- 
parently, while acting like a single disk. Usually, RAID is for 
redundancy—if one disk dies, your files are still intact. Other 
types of RAID increase the performance of storage. 


A bunch of disks in a RAID arrangement is called a RAID 
array. The type of RAID, called the RAID level, determines how 
many drive failures the array can tolerate and still guarantee the 
data’s safety. Some standard RAID levels are RAID-0, RAID-1, 
RAID-5, RAID-10, and others you can explore on the web. 


Let’s create a minimal RAID-1 array using the most common 
RAID software for Linux, mdadm. RAID-1 adds redundancy 
simply by mirroring data from one drive to the others in the 
array. As long as one drive is still operating, the data is safe. 
For this example, I start with two disks, /dev/sdf and /dev/sdg, 
each of which has a 10 GB partition, /dev/sdfl and /dev/sdg1. 
The steps I show are largely the same for other RAID levels and 
additional devices. For full details, visit the Linux Raid Wiki. 


WARNING 


RAID operations can wipe out filesystems without confir- 
mation. Practice the commands on spare drives or a virtual 
machine for safety. 


Create a RAID Array 
First, show that no RAID setup exists yet: 


> cat /proc/mdstat 
Personalities : No RAID types listed 


Create the RAID-1 array /dev/md1, from the two partitions: 
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> sudo mdadm --create /dev/md1 --level 1 \ 
--raid-devices 2 /dev/sdf1 /dev/sdg1i 


View /proc/mdstat again. The Personalities line now shows that 
RAID-1 is in use, and the next line shows the new array, md1, 
which is being built: 


> cat /proc/mdstat 
Personalities : [raid1] 
md1 : active raid1 sdg1i[1] sdf1[0] 
10474496 blocks super 1.2 [2/2] [UU] 
[SSS ess a: ceca Suaew aude ] resync = 45.8% ... 
finish=0.4min ... 


Optionally, wait for the build (“resync”) to complete: 


> cat /proc/mdstat 

Personalities : [raid1] 

mdi : active raid1 sdg1i[1] sdf1[0] 
10474496 blocks super 1.2 [2/2] [UU] 


However, you don't have to wait. The array is usable immedi- 
ately. Format and mount it like any other storage device: 


> sudo mke2fs /dev/md1i Format the array 
> sudo mkdir /mnt/raid Mount it 

> sudo mount /dev/mdi /mnt/raid 

> df -h /mnt/raid View it 
Filesystem Size Used Avail Use% Mounted on 
/dev/md1 9.9G 24K 9.4G 1% /mnt/raid 


Run \sb1k to illustrate the RAID configuration: 


> Usblk 
sdf 8:80 0 10G 0 disk 
ucsdf1 8:81 0 10G © part 
Lmd1 9:1 0 10G 0 raid1 /mnt/raid 
sdg 8:96 0 10G 0 disk 
L-sdg1 8:97 0 10G 0 part 
Lmd1 9:1 0 10G 0 raid1 /mnt/raid 


Run mdadm to see more details about the array: 
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> sudo mdadm --detail /dev/mdi 
dev/md1: 


Creation Time : Thu Jul 20 13:15:08 2023 
Raid Level : raid1 
Array Size : 10474496 (9.99 GiB 10.73 GB) 
Raid Devices : 2 
State : clean 
Working Devices : 2 


Number Major Minor RaidDevice State 
0 8 81 0 active sync /dev/sdf1 
1 8 97 1 active sync /dev/sdg1 


When you're satisfied with the array, save its configuration so it 
will survive reboots and mount itself. Don’t skip any steps. 


1. Save the RAID configuration to a file: 
> sudo mdadm --detail --scan --verbose > /tmp/raid 
> cat /tmp/raid 
ARRAY /dev/md1 level=raid1 num-devices=2 ... 


2. Use a text editor to append the contents of /tmp/raid to the 
configuration file /etc/mdadm/mdadm.conf, replacing any 
previous RAID configuration. 


3. Run this command to update the Linux kernel: 
> sudo update-initramfs -u 


4. Reboot. Check that your RAID array survived by mount- 
ing it by hand:? 
> sudo mount /dev/md1 /mnt/raid 
5. If everything worked, add this line to /etc/fstab so your 


RAID array mounts at boot time: 
/dev/md1 /mnt/raid ext4 defaults 0 2 


3 If your RAID array mysteriously renames itself /dev/md127, you for- 
got to run update-initramfs in the previous step. 
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WARNING 


Don’t update /etc/fstab too early. If your RAID configura- 
tion has a problem and you reboot, the computer might 
hang. Instead, test the configuration first by rebooting and 
mounting the array by hand, as I have. 


Replace a Device in a RAID Array 


So, your RAID array is up and running. What happens when a 
device dies and needs replacement? First, the failure is visible 
in /proc/mdstat. The failed device, /dev/sdf1, is marked with 
(F), and the uptime indicator, which should read [UU] (two 
devices up), reads [U_] (first device up, second device down). 


> cat /proc/mdstat 

Personalities : [raid1] 

md1 : active raid1 sdf1[1](F) sdg1[0] 
10474496 blocks super 1.2 [2/1] [U_] 


mdadm also shows the array as “degraded” and the device as 
“faulty”: 


> sudo mdadm --detail /dev/md1 
/dev/md1: 


Raid Devices : 2 
State : clean, degraded 

Working Devices : 1 

Failed Devices : 1 


Number Major Minor RaidDevice State 
1 8 97 - faulty /dev/sdf1 


To replace device /dev/sdf1, mark it as failed (if it isn't already) 
and remove it from the RAID array: 


> sudo mdadm --manage /dev/md1 --fail /dev/sdf1 
> sudo mdadm --manage /dev/md1 --remove /dev/sdfi 
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Shut down the computer, unplug the power cable, and physi- 
cally swap the failed storage device for a new one of the same 
size or larger. Pll call it by a nonexistent name /dev/NEW to 
clearly distinguish it in my instructions because the following 
commands are destructive, and you don’t want to mix up your 
drives. Substitute the correct device name on your system. 


Boot the computer, identify a good drive in the RAID array (in 
our case, /dev/sdg), and copy its partition table onto the new 
device with the sgdisk command. 


> sudo sgdisk -R /dev/NEW /dev/sdg = Copy from sdg to NEW 
> sudo sgdisk -G /dev/NEW Randomize GUIDs 


The device /dev/NEW now has a 10 GB partition, /dev/NEW1. 
Add it to the array: 


> sudo mdadm --manage /dev/md1 --add /dev/NEW1 
mdadm: added /dev/NEW1 


The array immediately begins rebuilding itself, mirroring data 
onto the new device: 


> cat /proc/mdstat 


SSSSSSSSSS>.....0000.] recovery = 51.5% ... 
finish=0.4min ... 


When mirroring is complete, the new device /dev/NEW1 has 
replaced the faulty device /dev/sdf1: 


> cat /proc/mdstat 

Personalities : [raid1] 

md1 : active raidi NEW1[1] sdg1i[0] 
10474496 blocks super 1.2 [2/2] [UU] 


Destroy a RAID Array 


Should you ever want to destroy the RAID array and use the 
partitions for other purposes, run these commands, assuming 
your device names are /dev/sdg1 and /dev/sdh1: 


> sudo umount /mnt/raid 
> sudo mdadm --stop /dev/md1 
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mdadm: stopped /dev/md1 
> sudo mdadm --zero-superblock /dev/sdg1 /dev/sdh1 


Finally, update /etc/fstab and /etc/mdadm/mdadm.conf to 
remove the RAID array /dev/md1, and inform the kernel that 
the array is gone: 


> sudo update-initramfs -u 


Logical Volumes for Flexible Storage 


pvcreate Create a physical volume. 

pvdispLay View details of physical volumes. 
pvremove _ Deletea physical volume. 

pvs View other details of physical volumes. 
vgcreate Create a volume group. 

vgdisplay View details of volume groups. 
vgextend Add physical volumes to a volume group. 
vgreduce — Remove physical volumes from a volume group. 
vgremove _ Delete a volume group. 

lvcreate — Createa logical volume. 

lvdisplay View details of logical volumes. 
lvresize — Resize a logical volume. 

lvremove _ Deletea logical volume. 


Logical volume management (LVM) solves two annoying prob- 
lems with disk storage: 


e Limited size. When a disk fills up, you have to delete files 
or replace it with a larger disk. 


e Fixed partitions. When you partition a disk, you guess 
how much space each partition will require, and if you're 
wrong, it’s time-consuming to change. 
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LVM solves these problems by wrapping a layer of abstraction 
around physical storage. It collects together a bunch of physical 
disks of any sizes, called physical volumes, to simulate one big 
disk, which it calls a volume group. The volume group becomes 
a playground for creating simulated partitions, called logical 
volumes, that can grow and shrink on demand. Figure 4-1 
shows the relationship between physical volumes (PVs), the 
volume group (VG) that contains them, and logical volumes 
(LVs) that you carve out of the total space. If the VG runs 
out of space, simply add another physical volume and the VG 
grows. If a partition (LV) is the wrong size, just change it. 
Existing files are preserved. Any mass storage devices can be 
part of a volume group, even RAID arrays created with mdadm 
(see “RAID Arrays for Redundancy” on page 224). 


LV LV 
/dev/myvg/stuff /dev/myvg/tiny 


/dev/sddi 


Figure 4-1. LVM concepts. Physical volumes (PV) are collected into a 
volume group (VG). Logical volumes (LV) are carved out of the VG. 


The most popular LVM software for Linux is called lvm2. It 
includes over 50 commands, which might seem like a lot, but 
their names follow a simple pattern: pv, vg, or lv, followed 
by a verb like create, remove, or display. So vgcreate creates 
a volume group, and pvdisplay prints information about a 
physical volume. 
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WARNING 


LVM operations can wipe out filesystems without confir- 
mation. Practice the commands on spare drives or a virtual 
machine for safety. 


Also, LVM provides no inherent redundancy. If one phys- 
ical volume dies, you lose the whole volume group. For 
more safety, run LVM on top of RAID; see “RAID Arrays 
for Redundancy” on page 224. 


I now present examples of using the most common lvm2 
commands, using three empty 10 GB disk partitions, /dev/ 
sdb1, /dev/sdcl, and /dev/sdd1.4 lvm2 has dozens more com- 
mands, however. For a full list, view the manpage of any lvm2 
command and jump to the end or visit sourceware.org/lvm2. 


Create a First Logical Volume 


This sequence of steps sets up two physical volumes, groups 
them into a 20 GB volume group, myvg, and creates a 15 GB 
logical volume, stuff: 


> sudo pvcreate /dev/sdbi /dev/sdci Create two PVs 
Physical volume "/dev/sdb1" successfully created. 
Physical volume "/dev/sdc1" successfully created. 

> sudo vgcreate myvg /dev/sdb1 /dev/sdc1  Createa VG 
Volume group "myvg" successfully created 


> sudo lvcreate -L 15G -n stuff myvg Create the LV 
Logical volume "stuff" created. 

> sudo pvs View the PVs 
PV VG Fmt Attr PSize PFree 
/dev/sdb1 myvg lvm2 a-- <10.00g 0 


/dev/sdc1 myvg lvm2 a-- <10.00g 5.34g 


4 Operate on partitions rather than whole disks. 
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The logical volume stuff is usable like any other storage device. 
Format and mount it: 


> sudo mke2fs /dev/myvg/stuff Format the LV 
> sudo mkdir /mnt/stuff Mount it 

> sudo mount /dev/myvg/stuff /mnt/stuff 

> df -h /mnt/stuff View it 
Filesystem Size Used Avail Use% Mounted on 


/dev/mapper/myvg-stuff 15G 24K 1.4G 1% /mnt/stuff 


When your LV is ready, add this line to /etc/fstab to mount it at 
boot time: 


/dev/mapper/myvg-stuff /mnt/stuff ext4 defaults 0 2 


View LVM Details 


The pvdisplay, vgdisplay, and lvdisplay commands print 
details about physical volumes, volume groups, and logical 
volumes, respectively. The commands pvs, vgs, and lvs print 
helpful summaries of that information. 


> sudo pvdisplay Show all PVs 

> sudo pvdisplay /dev/sdb1 Show selected PVs 
> sudo pvs Summarize PVs 
> sudo vgdisplay Show all VGs 

> sudo vgdisplay myvg Show selected VGs 
> sudo vgs Summarize VGs 
> sudo lvdisplay Show all LVs 

> sudo lvdisplay myvg/stuff = Show selected LVs 
> sudo lvs Summarize LVs 


Add a Logical Volume 


Let’s run lvcreate again to add a 2 GB logical volume called 
tiny to our volume group: 


> sudo lvcreate -L 2G -n tiny myvg 

Logical volume "tiny" created. 
> sudo mke2fs /dev/myvg/tiny Format 
> sudo mkdir /mnt/tiny Mount 
> sudo mount /dev/myvg/tiny /mnt/tiny 
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> df -h /mnt/tiny View 
Filesystem Size Used Avail Use% Mounted on 
/dev/mapper/myvg-tiny 2.0G 24K 1.9G 1% /mnt/tiny 


Add Disks to a Volume Group 


The vgextend command adds physical volumes to a volume 
group. Suppose you want to increase the size of stuff by 10 
GB (to 25 GB), but there’s only 3 GB of space left in volume 
group myvg. Enlarge your VG by adding a third physical vol- 
ume, /dev/sdd1, increasing the VG’s total size to 30 GB: 


> sudo pvcreate /dev/sddi Create another PV 
> sudo vgextend myvg /dev/sdd1 Grow the VG 


At this point, the LVM setup looks like Figure 4-1. Run lsblk 
to illustrate the LVM configuration: 


> Usblk /dev/sdb /dev/sdc /dev/sdd 


NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS 

sdb 8:16 0 10G 0 disk 

L-sdb1 8:17 © 10G O part 
Lmyvg-stuff 253:0 0 15G 0 lvm /mnt/stuff 

sdc 8:32 © 10G 0 disk 

L-sde1 8:33 © 10G @ part 
|-myvg-stuff 253:0 0 15G 0 lvm /mnt/stuff 
Lmyvg-tiny 253:1 0 2G 0 lvm /mnt/tiny 

sdd 8:48 0 10G 0 disk 

isdd1 8:49 © 10G 0 part 


Enlarge a Logical Volume 


The lvresize command grows or shrinks a logical volume.° 
Let’s enlarge the LV stuff to 25 GB: 


> sudo lvresize --resizefs --size 25G /dev/myvg/stuff 
> df -h /mnt/stuff 


5 Old-timers may resize an LV by running lvextend, umount, fsck, 
resize2fs, and mount in sequence. lvresize is easier. 
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Filesystem Size Used Avail Use% Mounted on 
/dev/mapper/myvg-stuff 25G 24K 24G 1% /mnt/stuff 


Shrink a Logical Volume 


It turns out the LV stuff doesn’t need to be so large. Shrink it to 
8 GB (making sure first that it has less than 8 GB in use): 


> sudo lvresize --resizefs --size 8G /dev/myvg/stuff 
Do you want to unmount "/mnt/stuff" ? [Y|n] y 


Logical volume myvg/stuff successfully resized. 

> df -h /mnt/stuff 

Filesystem Size Used Avail Use% Mounted on 
/dev/mapper/myvg-stuff 7.9G 24K 7.5G 1% /mnt/stuff 


Delete a Logical Volume 


The lvremove command deletes a logical volume. Let’s get rid 
of the LV tiny: 


> sudo umount /mnt/tiny First unmount the LV 
> sudo lvremove /dev/myvg/tiny Remove the LV 
Do you really want to remove and DISCARD 
active logical volume myvg/tiny? [y/n]: y 
Logical volume "tiny" successfully removed 


Reduce a Volume Group 


The vgreduce command removes an unused physical volume 
from a volume group. The PV remains managed by lvm2, just 
not within the VG myvg: 


> sudo vgreduce myvg /dev/sdd1 
Removed "/dev/sdd1" from volume group "myvg" 


Delete a Volume Group 


The vgremove command removes a volume group and deletes 
any logical volumes it contains: 


> sudo vgremove myvg 
Do you really want to remove volume group "myvg" 
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containing 1 logical volumes? [y/n]: y 

Do you really want to remove and DISCARD 

active logical volume myvg/stuff? [y/n]: y 
Logical volume "stuff" successfully removed 
Volume group "myvg" successfully removed 


Delete a Physical Volume 
The pvremove command removes physical devices from LVM: 


> sudo pvremove /dev/sdb1 /dev/sdci 
Labels on physical volume "/dev/sdb1" wiped. 
Labels on physical volume "/dev/sdc1" wiped. 


ZFS: A Modern, Do-It-All Filesystem 


zpool Configure a ZFS storage pool. 
zfs Configure a ZFS dataset. 


WARNING 


ZFS operations can wipe out filesystems without confirma- 
tion. Practice the commands on spare drives or a virtual 
machine for safety. 


ZFS (Zettabyte File System) packs advanced features like RAID, 
logical volume management, encryption, and compression into 
one convenient package. If you're accustomed to traditional 
Linux filesystems like ext4, ZFS may seem like an alien world. 
It has its own terminology with “pools” and “vdevs.” It doesn’t 
use /etc/fstab or the mount command. You don’t even need to 
partition or format your disks explicitly. 


A ZFS vdev, short for “virtual device,” is a group of physical 
disks that work together. They might divide the data among 
themselves as if they were one big disk, like a RAID-0 “disk 
striping” setup. They might mirror each other for redundancy, 
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like a RAID-1 setup. They might operate as a disk cache; and 
there are other possibilities. 


A collection of vdevs is called a pool. A pool acts like one big 
storage device. You can carve it up into units that are sort 
of like partitions, called datasets, and you can change their 
size limits and other attributes flexibly. Dataset names look 
like Linux paths without a leading slash. For example, a pool 
named mypool with a dataset named stuff would be named 
mypool/stuff. (Datasets can contain other datasets too, like 
mypool/stuff/important.) Add a leading slash, and you get the 
dataset’s default Linux mount point, like /mypool/stuff. 


ZFS isn't the only filesystem with advanced capabilities— 
another popular one is Btrfs—but it’s among the easiest to 
configure. 


NOTE 


I discuss only the minimal ZFS functionality to do inter- 
esting things. Real ZFS systems need careful configura- 
tion, tuning, and plenty of RAM; read the docs at https:// 
oreil.ly/-t5Fu. 


To demonstrate ZFS with both mirroring and striping, I use 
two pairs of disks, as in Figure 4-2. Each pair is a vdev with 
mirroring (RAID-1). ZFS then stripes across the two vdevs 
(RAID-0), effectively creating a RAID-10 setup. This redun- 
dant pool can tolerate one failed drive in each vdev and keep 
the data safe. 
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Striping 
(RAID-O) 


Mirroring vdev Mirroring vdev 
(RAID-1) (RAID-1) 


es) (Ee) Ge) Be 
/dev/sdb /dev/sdc /dev/sdd /dev/sde 


Figure 4-2. Our example ZFS configuration: a RAID-10 pool 


Create a ZFS Pool 


Use the zpool command to construct the pool of two pairs of 
mirrored drives. I create a pool called mypool from four 10 GB 
disk devices, /dev/sdb, /dev/sdc, /dev/sdd, and /dev/sde: 


> sudo zpool create mypool \ 
mirror /dev/sdb /dev/sdc \ 
mirror /dev/sdd /dev/sde 


WARNING 


The simple device names in my examples, like /dev/sdb, 
can change after a reboot. For a more robust setup, use 
names that are guaranteed not to change, like the symbolic 
links found in /dev/disk/by-id or /dev/disk/by-uuid. 


Also, on real systems, be sure to set an appropriate align- 
ment shift value with -o ashift on creation; see the docs. 
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Use zpool status to view the results. 


> zpool status 
pool: mypool 
state: ONLINE 


config: 

NAME STATE READ WRITE CKSUM 
mypool ONLINE 0 0 0 = The pool 
mirror-@ ONLINE 0 0 0 First vdev 

sdb ONLINE 0 0 0 

sdc ONLINE 0 0 0 

mirror-1 ONLINE 0 0 0 Second vdev 
sdd ONLINE 0 0 0 
0 0 (0) 


sde ONLINE 


Create a ZFS Dataset 


Traditional filesystems have partitions of fixed size that you 
mount in the file /etc/fstab. ZFS has datasets of arbitrary size 
that it mounts automatically. Create a dataset named data in 
pool mypool, mounted at the directory /mypool/data: 


> sudo zfs create -o mypool/data 
Move the mount point if you like, to /mnt/stuff: 

> sudo zfs set mountpoint=/mnt/stuff mypool/data 
View the results with either of these commands: 


> zfs mount 

mypool /mypool The whole pool 
mypool/data /mnt/stuff Your dataset 
> zfs get mountpoint mypool/data 

NAME PROPERTY VALUE SOURCE 
mypool/data mountpoint /mnt/stuff local 


Now use the dataset like any other mounted partition: 


> sudo cp /etc/hosts /mnt/stuff 
> cd /mnt/stuff 

> ls 

hosts 
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Create an Encrypted ZFS Dataset 


By adding a few options, you can create a dataset that’s encryp- 
ted and requires a passphrase before mounting. Create an 
encrypted dataset named mypool/cryptic: 


> zfs create \ 
-o encryption=on \ 
-o keylocation=prompt \ 
-o keyformat=passphrase \ 
mypool/cryptic 
Enter new passphrase: XxXxXXxXXXX 
Re-enter new passphrase: xxXxXXxXxXXxxX 


Use the dataset normally. When you reboot or otherwise need 
to mount the dataset, run: 


> sudo zfs mount -l mypool/cryptic 


Set Size Limits on ZFS Datasets 


By default, a ZFS dataset is the same size as the pool. Limit its 
size by setting a quota, which you may change anytime: 


> sudo zfs set quota=15g mypool/data 


> zfs list 
NAME USED AVAIL REFER MOUNTPOINT 
mypool 312K 18.4G 25K /mypool 


mypool/data 24K 15.0G 24K /mnt/stuff 15 GB limit 


Enable Compression on ZFS Datasets 


ZFS can automatically compress data as it's written and uncom- 
press it when read. It supports various compression algorithms; 
here I use gzip compression and view how effectively files are 
being compressed (the compression ratio): 


> sudo zfs set compression=gzip mypool/data 


> cp hugefile /mnt/stuff Store a big file 

> zfs get compressratio See the compression ratio 
NAME PROPERTY VALUE SOURCE 
mypool compressratio 122.66x - 


mypool/data compressratio 126.12x - 
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After enabling compression, only new data is compressed; 
existing files are not. To turn off compression: 


> sudo zfs set compression=off mypool/data 


Snapshot a ZFS Dataset 


ZFS supports snapshots: storing the state of a dataset so you can 
easily return to that state (roll back) later. Before performing a 
risky change on your files, for example, take a snapshot, and if 
something goes wrong, you can revert the change with a single 
command. Snapshots occupy very little disk space, and you can 
send them efficiently to other zpools or hosts (with zfs send 
and zfs recv). Create a snapshot of mypool/data named safe: 


> sudo zfs snapshot mypool/data@safe 
List your snapshots: 


> zfs list -t snapshot 
NAME USED AVAIL REFER MOUNTPOINT 
mypool/data@safe OB - 24K OC 


If you can't list snapshots, set listsnapshots=on and try again: 
> sudo zpool set listsnapshots=on mypool 


Try changing some files in mypool/data. Then roll back to the 
snapshot safe and see that your changes are gone: 


> sudo zfs rollback mypool/data@safe 


Destroy a ZFS Dataset or Snapshot 


Be careful with the zfs destroy command—it runs immedi- 
ately without confirmation. 


> sudo zfs destroy mypool/data@safe A snapshot 
> sudo zfs destroy mypool/data A dataset 
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Destroy a ZFS Pool 


Be careful with the zpool destroy command—it runs immedi- 
ately without confirmation. 


> sudo zpool destroy mypool 


Backups and Remote Storage 


rsync Efficiently copy a set of files, even across a network. 
rclone Sync files with various cloud providers. 
dd Low-level copying of data. 


growisofs Burna DVD or Blu-ray disc. 
You can back up your precious Linux files in various ways: 


e Copy them to a remote machine. 
e Copy them to a backup medium like an external drive. 


e Burn them onto a disc. 


I present a few popular Linux commands for backups, but there 
are others. Some users prefer cpio for its flexibility, and some 
long-time administrators swear by dump and restore as the only 
reliable way to back up and restore every type of file. See the 
manpages for these commands if you are interested in them. 


rsync stdout -opt --help — --version 
rsync [options] source destination 


The rsync command copies a set of files. It can make an exact 
copy, including file permissions and other attributes (called 
mirroring), or it can just copy the data. It can run over a net- 
work or on a single machine. rsync has many uses and over 50 
options; I present just a few common cases relating to backups. 
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To mirror the directory mydir and its contents into another 
directory mydir2 on a single machine: 


> rsync -a mydir mydir2 


rsync is finicky about how you specify the first directory. If 
you write “mydir” as in the example here, that directory is 
copied into mydir2, creating the subdirectory mydir2/mydir. If 
instead, youd rather have only the contents of mydir copied 
into mydir2, append a slash onto “mydir”: 


> rsync -a mydir/ mydir2 


rsync can mirror a directory over a network to another host, 
securing the connection with SSH to prevent eavesdropping. 
Here I copy directory mydir to the account “smith” on remote 
host server.example.com, in a directory D2: 


> rsync -a mydir smith@server.example.com:D2 


If you like rsync but also want to have incremental back- 
ups and manage them efficiently, try rsnapshot (https://oreil.ly/ 
VNfb-). 


Useful options 


-o Copy the ownership of the files. (You might need superuser privileges on the 
remote host.) 


-g Copy the group ownership of the files. (You might need superuser privileges 
on the remote host.) 


-p Copy the file permissions. 

-t Copy the file timestamps. 

-r Copy directories recursively (i.e., including their contents). 

-L_ Permit symbolic links to be copied (rather than the files they point to). 
-D Permit devices to be copied. (Superuser only.) 


-a_ Mirroring: copy all attributes of the original files. This implies all of the 
options -Dogptr1 (think “dog patrol”). 
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-x When copying a tree of files, remain within the current filesystem; do not 
cross over into other mounted filesystems. 


-z Compress the data for transit. Only useful for remote hosts over slow 
connections. Avoid compression when copying files locally (it’s wasteful). 


-n_ Dry-run mode: don’t actually copy. Just display what would be done. 


-v_ Verbose mode: print status information during the copy. Add - - progress 
to display a numeric progress meter while files are copied. 


rclone --opt --help — --version 
rclone subcommand [options] [arguments] 


The rclone command connects your Linux system to popular 
cloud storage providers to copy files conveniently. It works 
with Dropbox, Google Drive, Microsoft OneDrive, Amazon S3, 
and about 50 other destinations. To get started, run rclone 
config and follow the prompts to set up a connection with 
your cloud provider of choice, which rclone calls a remote. 
Visit rclone.org/docs for detailed instructions on configuration. 


After choosing a name for your remote, such as myremote, refer 
to remote files with the syntax myremote: path, where path is a 
Linux-style file path. For example, a file photo.jpg in a remote 
directory Photos would be myremote:Photos/photo. jpg. 


Backups are usually done with the rclone sync command, 
which synchronizes a local directory and a remote directory so 
they contain the same content, adding or deleting as necessary. 
It works much like the rsync --delete command. You can 
synchronize in either direction, from your local machine to the 
remote, or from the remote to your local machine. You can 
even set up client-side encryption, so files are transparently 
encrypted before they're copied to the remote and decrypted 
when they’re copied back to your local system (see the docs). 


Some common operations for backups include: 
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rclone ls remote: List files on the remote recursively 
(append --max-depth 1 for 


no recursion). 

rclone lsd remote: List only directories on the 
remote. 

rclone lsl remote: Display along listing like Ls 


-1 from the remote recursively 
(append --max-depth 1 for 


No recursion). 
rclone copy myfile remote: Copy a local file to the remote. 
rclone copy remote:myfile . Copy a remote file to your local 
system. 
rclone move myfile remote: Move a local file to the remote. 
rclone move remote:myfile . Move a remote file to your local 
system. 
rclone delete remote:myfile Delete a remote file. 


rclone sync mydir remote:mydir Synchronize local files onto the 
remote. 


rclone sync remote:mydir mydir Synchronize remote files onto 
your local system. 


Run rclone help for a complete list of subcommands. Note 
that you can access fancier paths on the remote than my earlier 
examples show: 


> rclone copy remote:Photos/Vacation/picture.jpg . 
Useful options 


-n_ Dry-run mode: don’t actually copy. Just display what would be done. 


-i  Runinteractively so you're prompted before changes. 
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dd stdin stdout --help — --version 
dd [options] 


dd is a low-level copier of bits and bytes. It can copy data from 
one file to another, say, from myfile to /tmp/mycopy: 


> dd if=myfile of=/tmp/mycopy 

2+1 records in 

2+1 records out 

1168 bytes (1.2 kB) copied, 0.000174074 s, 6.7 MB/s 


It can even convert data while copying, like changing charac- 
ters to uppercase while copying from myfile to /tmp/mycopy: 


> dd if=myfile of=/tmp/mycopy conv=ucase 


dd does much more than copy files. It can copy raw data 
directly from one disk device to another. Here's a command 
to clone a hard disk: 


> sudo dd if=/dev/device1 of=/dev/device2 bs=512 \ 
conv=noerror, sync OVERWRITES /dev/device2 


Or, copy an entire disk device to create an ISO file. Make sure 
the output file is on a different disk device and has sufficient 
free space. 


> sudo dd if=/dev/device of=disk_backup.iso 


WARNING 


dd, when run as the superuser, can wipe out your hard 
drive in seconds if youre not careful. Always double-check 
that the output file (the argument of=) is the one you 
intend. Back up your computer and keep a Linux “live” 
distro on hand for emergencies (see “What’s in This 
Book?” on page ix) before experimenting with dd as root. 


Backups and Remote Storage | 245 


For some great advice on sophisticated uses of dd, visit https:// 
oreil.ly/R3krx. My favorite tip is backing up a disk’s master boot 
record (MBR), which is 512 bytes long, to a file called mbr.txt: 


> sudo dd if=/dev/device of=mbr.txt bs=512 count=1 


Useful options 


if=file Specify aninput file or device. 


of=file Specify an output file or device. Double-check that it’s the 
correct one! 


bs=N Copy N bytes (the “block size”) at a time. (To set different block 
sizes for the input and output, use ibs and obs, respectively.) 


skip=N Skip past WV blocks of input (of size ibs) before starting the copy. 
seek=N Discard N blocks of output (of size obs) before starting the copy. 


conv=spec Convert the data being copied. spec can be ucase (convert to 
uppercase), Lcase (convert to lowercase), ascii (convert to 
ASCII from EBCDIC), and many others listed on the manpage. 


growisofs stdin -file --opt --help  --version 


growisofs [options] tracks 


The growisofs command burns a writable CD, DVD, or Blu- 
ray disc. To burn the contents of a Linux directory onto a disc 
readable on Linux, Windows, and macOS systems: 


1. Locate your disc writer devices by running: 
> grep "4drive name:" /proc/sys/dev/cdrom/info 
drive name: sri sr0 


The available devices here are /dev/sr1 and /dev/sr0. 


2. Put the files you want to burn into a directory, say, dir. 
Arrange them exactly as youd like on the disc. The direc- 
tory dir itself is not copied to the disc, just its contents. 
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3. Use the mkisofs command to create an ISO (disc) image 
file, and burn it onto a disc using growisofs, assuming 
your device is /dev/sr1: 

> mkisofs -R -l -o SHOME/mydisk.iso dir 
> growisofs -dvd-compat -Z /dev/sr1=$HOME/mydisk.iso 
> rm SHOME/mydisk.iso 


To burn audio CDs, use a friendly graphical program like k3b. 
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CHAPTER 5 
Networking Commands 


Host Information 


uname Print basic system information. 
hostname Print the system’s hostname. 
ip Set and display network interface information. 


Every Linux machine or host has a name, a network IP address, 
and other properties. Here’s how to display this information. 


uname stdout -opt --help — --version 
uname [options] 


The uname command prints fundamental information about the 
OS, particularly the kernel: 


> uname -a 

Linux myhost 5.15.0-76-generic #83-Ubuntu 
SMP Thu Jun 15 19:16:32 UTC 2023 x86_64 
x86_64 x86_64 GNU/Linux 


This includes the kernel name (Linux), hostname (myhost), 
kernel release (5.15.0-76-generic), kernel version (#83-Ubuntu 
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SMP Thu Jun 15 19:16:32 UTC 2023), hardware name 
(x86_64), processor type (also x86_64), hardware platform 
(also x86_64) and OS name (GNU/Linux). 


Useful options 


-a_ Allinformation 

-s_ Only the kernel name (the default) 

-n_ Only the hostname, as with the hostname command 
-r Only the kernel release 

-v_ Only the kernel version 

-m Only the hardware name 

-p Only the processor type 

-i Only the hardware platform 


-o Only the 0S name 


hostname stdout -opt --help — --version 
hostname [options] [name] 


The hostname command prints the name of your computer. 
Depending on how you have things set up, this might be the 
fully qualified hostname: 


> hostname 
myhost.example.com 


or your short hostname: 


> hostname 
myhost 


You can also set your hostname, as root: 


> sudo hostname orange 
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NOTE 


Changes made by hostname might not survive a reboot. 
Consult your distro’s documentation for any additional 
steps to make the change permanent, such as editing a 
configuration file that is read at boot time. 


Useful options 


-s Print your host's short name. 

-f Print your host's fully qualified name. 

-d Print your host's DNS domain name. 

-a Print all of your host's aliases. 

-I Print all of your host’s network addresses. 


-F hostfile Set your hostname by reading it from file hostfile. 


Ip stdout ~-help 


ip [options] object command... 


The ip command displays and sets various aspects of your 
computer’s network interface. This topic is beyond the scope of 
the book, but I'll teach you a few tricks. 


If you know the name of your default interface, such as eth0, 
ens 160, or enp68s0, view its details with this command: 


> ip addr show etho 

2: ethO: <BROADCAST,MULTICAST,UP,LOWER_UP> ... 
link/ether 00:50:ba:48:4f:ba brd ff:ff:ff: ... 
inet 192.168.0.21/24 brd 192.168.0.255 scope ... 
inet6 fe80::21e:8cff:fe53:41e4/64 ... 


They include the IPv4 address (192.168.0.21), IPv6 address 
(fe80::21e:8cff:fe53:41e4/64), MAC address (00:50:ba:48:4f:ba), 
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and other details. If you don’t know the interface name, view all 
loaded network interfaces: 


> ip addr show | less 


Another command, ifconfig, presents information similar to 
ip addr show. It’s considered mostly obsolete but you'll still 
find it on many systems. 


> ifconfig etho Details of one network interface 
> ifconfig -a Details of all network interfaces 


Display other network information with the following ip com- 
mands. Add help on the end of any command (e.g., ip link 
help) for usage details. 

ip addr Display IP addresses of your network devices. 

ip maddr Display multicast addresses of your network devices. 

ip link Display attributes of your network devices. 

ip route Display your routing table. 

ip monitor Begin monitoring your network devices; press “C to stop. 

ip help See usage information for all these commands. 
Superusers can also use ip to configure network devices, man- 
age routing tables and rules, create tunnels, and more. Learn 


more about ip and related tools (known as iproute2) and on 
the ip manpage. 


Host Location 
host Look up hostnames, IP addresses, and DNS info. 
whois Look up the registrants of internet domains. 
ping Check if a remote host is reachable. 


traceroute View the network path to a remote host. 
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When dealing with remote computers, you might want to 
know more about them. Who owns them? What are the IP 
addresses? Where on the network are they located? 


host stdout -- opt 
host [options] name [server] 


The host command looks up the hostname or IP address of a 
remote machine by querying DNS: 


> host www.ubuntu.org 

www.ubuntu.com has address 69.16.230.226 

> host 69.16.230.226 
226.230.16.69.in-addr.arpa domain name pointer 
1b05.parklogic.com. 


It can also find out much more: 


> host -a www.ubuntu.org 

Trying "www.ubuntu.org" 

33 ->>HEADER<<- opcode: QUERY, status: NOERROR ... 
33 flags: qr rd ra; QUERY: 1, ANSWER: 2, 


33 QUESTION SECTION: 
j;www.ubuntu.org. IN ANY 


3; ANSWER SECTION: 
www.ubuntu.org. 60 IN CNAME ~ubuntu.org. 


though a full discussion of this output is beyond the scope 
of this book. The final, optional “server” parameter specifies a 
particular nameserver for the query: 


> host www.ubuntu.org nsi.parklogic.com 
Using domain server: 

Name: ns1.parklogic.com 

Address: 69.16.230.48#53 

Aliases: 

www.ubuntu.org has address 69.16.230.226 
www.Ubuntu.org mail is handled by ... 
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Useful options 


To see all options, run host by itself. 


-a Display all available information about a host. 


-t Choose the type of nameserver query: A, AXFR, CNAME, HINFO, KEY, MX, 
NS, PTR, SIG, SOA, and so on. 


Here’s an example of the -t option to locate MX records: 


> host -t MX centos.org 
centos.org mail is handled by 10 mail.centos.org. 


If the host command doesn't meet your needs, try dig, another 
DNS lookup utility, or the nslookup command, which is mostly 
obsolete but still around. 


whois stdout --opt --help — --version 
whois [options] domain_name 


The whois command prints the registration of an internet 
domain. The output is long, so pipe it through less: 


> whois Linuxmint.com | less 
Domain name: LINUXMINT.COM 
Registrar: Ascio Technologies, Inc. Danmark 
Updated Date: 2023-05-161T22:22:38Z 
Creation Date: 2006-06-07T10:45:34Z 
Name Server: NSO1.SERVAGE.NET 


Useful options 


-hserver — Perform the lookup at the given registrar's whois server. For 
example: 
> whois -h whois.comlaude.com redhat.com 


-p port Query the given TCP port instead of the default (43). 
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ping stdout -- opt 
ping [options] host 


The ping command tells you if a remote host is reachable. It 
sends small messages called ICMP packets to a remote host, 
waits for responses, and prints how long they took: 


> ping slackware.com 

PING slackware.com (64.57.102.36) 56(84) bytes of data. 
64 bytes from connie.slackware.com (64.57.102.36): 
icmp_seq=1 ttl=52 time=107 ms 

64 bytes from connie.slackware.com (64.57.102.36): 
icmp_seq=2 ttl=52 time=107 ms 

aC 

--- slackware.com ping statistics --- 

2 packets transmitted, 2 received, 0% packet loss, ... 
rtt min/avg/max/mdev = 107.207/107.350/107.494/0.143 ms 


Here, the host slackware.com replied in 107 milliseconds. 
(Notice that its real name is connie.slackware.com.) Pings may 
be blocked by firewalls, so a running host might not respond. 


Useful options 
-c N_ Ping at most N times. (Note that “ping’” is a verb.) 
-i N Wait N seconds (default 1) between pings. (Here “ping” is a noun.) 


-n Print IP addresses in the output, rather than hostnames. 


-6 Use IPv6 instead of the default IPv4. 


traceroute stdout --opt --help — --version 
traceroute [options] host [packet_length] 


The traceroute command prints the network path from your 
local host to a remote host, and the time it takes for packets to 
traverse the path: 
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> traceroute archlinux.org 
1 server.mydomain.com (192.168.0.20) 1.397 ms ... 
2 10.221.16.1 (10.221.16.1) 15.397 ms ... 
3  router.example.com (92.242.140.21) 4.952 ms ... 


12 archlinux.org (95.217.163.246) 117.100 ms ... 


traceroute sends three “probes” to each host in the path and 
reports the return times. If five seconds pass with no response, 
traceroute prints an asterisk. If traceroute is blocked by fire- 
walls or cannot proceed for other reasons, it prints a symbol: 


Symbol Meaning 


!F Fragmentation needed 

!H Host unreachable 

IN Network unreachable 

IP Protocol unreachable 

1S Source route failed 

1X Communication administratively prohibited 
IN ICMP unreachable code NV 


The default packet size is 40 bytes. Change this value with 
the final, optional packet_length parameter (e.g., traceroute 
myhost 120). For a more interactive experience, try the mtr 
command (“my traceroute”). 


Useful options 


-n Numeric mode: print IP addresses instead of hostnames. 
-w N_ Change the timeout from five seconds to N seconds. 


-6 Use IPv6 instead of the default IPv4. 


Network Connections 


ssh Log into a remote host, or run commands on it. 
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scp Copy files between two hosts. 
sftp __ Interactively copy files between two hosts. 
netcat Create arbitrary network connections. 


Linux makes it easy to connect from one host to another 
securely for remote logins, file transfers, and other purposes. 


ssh stdin stdout 


ssh [options] host [command] 


The ssh (Secure Shell) command securely logs you into a 
remote machine. If your local and remote usernames are the 
same, just provide the remote hostname: 


> ssh remote.example.com 
smith@remote.example.com's password: XXXxXXXXXX 


Otherwise, provide your remote username as well: 


> ssh sandy@remote.example.com One syntax 
> ssh -l sandy remote.example.com Another syntax 


If you provide a command, ssh invokes it on the remote 
machine without starting an interactive login session: 


> ssh sandy@remote.example.com df Check free disk space 


ssh encrypts all data that travels across its connection, includ- 
ing your username and password. The SSH protocol also sup- 
ports other ways to authenticate, such as public keys: see the 
sidebar “Public Key Authentication with SSH” on page 258. 


Useful options 


-Luser Specify your remote username; otherwise, ssh assumes it equals your 
local username. Or use the syntax username@host: 
> ssh sandy@remote.example.com 
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-pport Connect to a port number other than the default (22). 


-t Allocate a tty on the remote system; useful when trying to run a remote 
command with an interactive UI, such as a text editor. 


-V Produce verbose output, useful for debugging. Repeat - v for more 
detailed messages. 


Public Key Authentication with SSH 


Public key authentication is a more secure way to log into 
remote hosts with SSH. It uses a pair of cryptographic keys. 
Your private key remains on your local host in the directory 
~/.ssh, protected by a passphrase. Your public key is copied to 
remote hosts. First, generate a key pair: 


> ssh-keygen 
Enter file to save the key (~/.ssh/id_rsa): <Enter> 
Enter passphrase (empty for no passphrase): XXXXXXX 


Enter same passphrase again: XXXXXXX 
Your identification has been saved in ~/.ssh/id_rsa 


Then, copy your public key to the remote host and log in: 


> ssh-copy-id sandy@remote.example.com 
sandy@remote.example.com's password: XXXXXXXX 

Number of key(s) added: 1 

> ssh sandy@remote.example.com 

Enter passphrase for '/home/sandy/.ssh/id_rsa': XXXXXXX 


If the final ssh command fails to log you in, check that: 
e The local and remote ~/.ssh directories have mode 0700. 
¢ The local private key file ~/.ssh/id_rsa has mode 0600. 
¢ The remote file ~/.ssh/authorized_keys has mode 600. 


e The public key in local file ~/.ssh/id_rsa.pub appears in 
remote file ~/.ssh/authorized_keys. If not, append the local 
file’s contents to the remote authorized_keys and try again. 


Also run ssh -v to print debug messages and look for clues. 
Run ssh -vwv for more verbose detail. 
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scp stdout 
scp local_spec remote_spec 


The scp (secure copy) command copies files and directories 
from one computer to another in batch. (For an interactive 
UL, see sftp.) It encrypts all communication between the two 
machines using SSH. As a simple example, scp can copy a local 
file to a remote machine: 


> scp myfile remote.example.com:newfile 
recursively copy a directory to a remote machine: 
> scp -r mydir remote.example.com: 
copy a remote file to your local machine: 
> scp remote.example.com:myfile . 
or recursively copy a remote directory to your local machine: 
> scp -r remote.example.com:mydir . 


You can even copy files from one remote machine to another if 
you have SSH access to both machines: 


> scp remote1.example.com:myfile remote2.example.com: 


To specify an alternate username on the remote system, use the 
username@host syntax: 


> scp myfile sandy@remote.example.com: 


Useful options 


-P port Connect on the given TCP port number (default is 22). 


-p Preserve all file attributes (permissions, timestamps) when copying. 
-r Recursively copy directories and their contents. 
-v Produce verbose output, useful for debugging. 
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sftp stdin stdout --opt --help — --version 
sftp (host | username@host) 


The sftp command is an interactive program that copies files 
securely between two computers using SSH. (Avoid the simi- 
larly named command ftp—most implementations send user- 
names, passwords, and files insecurely over the network.) 


> sftp remote.example.com 
smith@remote.example.com's password: xXXXXXXX 
sftp> cd MyFiles Change remote directory 
sftp> ls List remote directory 
README 

file1 

file2 

file3 

sftp> get file2 Transfer a remote file 
Fetching /home/sandy/MyFiles/file2 to file2 
sftp> quit 


If your username on the remote system is different from your 
local one, use the username@host argument: 


> sftp sandy@remote.example.com 


Command Meaning 


help View a list of available commands. 

ls List the files in the current remote directory. 

lls List the files in the current local directory. 

pwd Print the remote working directory. 

lpwd Print the local working directory. 

cddir Change your remote directory to be dir. 

lcd dir Change your local directory to be dir. 

get file? [file2] Copy remote file7 to local machine, optionally 
renamed as file2. 
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Command Meaning 

put file! [file2| Copy local file7 to remote machine, optionally 
renamed as file2. 

mget file* Copy multiple remote files to the local machine 
using patterns * and ?. 

mput file* Copy multiple local files to the remote machine 
using patterns * and ?. 

quit Exit sftp. 

netcat stdin stdout 


netcat [options] [destination] [port] 


nc [options] [destination] [port] 


netcat, or equivalently, nc, is a general-purpose tool to make 
network connections. It’s handy for debugging, learning about 
networking, and many other uses. For example, netcat can 
speak directly to any TCP or UDP service, such as an SSH 
server (if one is running) on your local TCP port 22: 


> hetcat localhost 22 
SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntud.1 
AC 


This feature, which can determine if a particular service is 
up or down, also works with service names as listed in the 
file /etc/services. For example, you could connect to Google’s 
web service (port 80) with: 


> netcat www.google.com http 

abc <Enter> Type some junk and press Enter 
HTTP/1.0 400 Bad Request 

Content-Type: text/html; charset=UTF-8 

Content-Length: 1555 

Date: Tue, 18 Jul 2023 03:14:36 GMT 
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For old-school Linux users who run telnet to connect to TCP 
ports, netcat is more flexible. For example, create a client and a 
service that talk to each other. Begin with a service listening on 
port 55555: 


> netcat -1l 55555 


In another window, run a client that talks to that same port, 
and type a message: 


> netcat localhost 55555 
Hello world, how are you? <Enter> 


Your message is sent to your service, which prints “Hello world, 
how are you?” and any subsequent lines you enter. Press *C in 
the client to close the connection. 


Useful options 


-u Establish a UDP connection instead of TCP. 
-1 _ Listen for connections on the given port. 
-p N_ Use port Nas the source port. 

-w N_ Time out after NV seconds. 


-h Get help. 


Email in Daily Use 


mutt — Text-based mail client. 
mail — Minimal text-based mail client. 


mailq View the outgoing mail queue on your system. 


Most users receive their email in the cloud and read it in a 
web browser or a graphical email application. Few people read 
their email at the command line. Nevertheless, text-based email 
programs have interesting uses, particularly in scripts. I present 
one full-featured email client (mutt) that runs on the command 
line, followed by a few other mail-related commands. I assume 
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your system is already configured to accept email; otherwise, 
some commands will not work. 


mutt stdin stdout 
mutt [options] 


Mutt is a text-based mailer that runs in a shell, so it can be used 
both locally (e.g., in a terminal window) or remotely over an 
SSH connection. To invoke it, run: 


> mutt 


When the main screen appears, mutt briefly lists messages in 
your mailbox, one per line. Try the following operations: 


Up arrow Move to the previous message. 

Down arrow Move to the next message. 

PageUp Scroll up one pageful of messages. 

PageDown Scroll down one pageful of messages. 

Home Move to the first message. 

End Move to the last message. 

m Compose a new mail message. This invokes your default 


text editor. After editing the message and exiting the 
editor, type y to send the message or q to postpone it. 


r Reply to current message. Works like m. 

f Forward the current message. Works like m. 

i View the contents of your mailbox. 

Cc Copy the current message to another mailbox. 
d Delete the current message. 


When you've finished editing a message, try these operations: 
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Keystroke Meaning 


a Attach a file (an attachment) to the message. 
c Set the CC list. 

b Set the BCC list. 

e Edit the message again. 

r Edit the Reply-To field. 

s Edit the subject line. 

y Send the message. 

Cc Copy the message to a file. 

q Postpone the message without sending it. 


The following operations are always available: 


Keystroke Meaning 


? See a list of all commands (press the space bar to scroll 
down, q to quit). 
AG Cancel the command in progress. 
q Quit. 
mail stdin stdout 


mail [options] recipient 


The mail command is a simple email client. It's best for quick 
messages from the command line or in scripts. Send a message: 


> mail smith@example.com 

Subject: my subject 

I'm typing a message. 

To end it, I type a period by itself on a line. 


Cc: jones@example.com 
+ 
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Pipe any command’s output to mail to send it in an email 
message. This feature is particularly useful in shell scripts. 


> echo "Wake up!" | mail -s "Alert" smith@example.com 


To mail the contents of a text file, simply redirect the file to 
mail. This won't work for binary files like images, which must 
be converted to attachments first. 


> mail -s "my subject" smith@example.com < file.txt 
Useful options 


-s subject Set the subject line of an outgoing message. 
-c addresses (CCtothe given addresses, a comma-separated list. 


-b addresses BCCto the given addresses, a comma-separated list. 


-v Verbose mode: print messages about mail delivery. 
mailq stdout 
mailq 


The mailq command lists outgoing email messages awaiting 
delivery, if any. 


. Size-- ---- Arrival Time-- -Sender/Recipient--- 
333 Tue Jan 10 21:17:14 smith@example.com 
jones@elsewhere.org 


Mail delivery is often so quick that mailq has no output. 
Sent messages are also recorded in a log file such as /var/log/ 
mail.log. The name may differ from distro to distro. View the 
last few lines with tail: 


> tail /var/log/mail. log 
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Beyond Mail Readers 


Email is more “transparent” on Linux than on other platforms 
that merely display your mailbox and send and receive mes- 
sages. The ability to list outgoing email messages with mailgq is 
just one example. Here are some other options to whet your 
appetite and encourage you to explore. 


e Process your mailboxes with any command-line tools, 
such as grep, because mail files are plain text. 


« Manually retrieve messages from a remote mail server at 
the command line with the fetchmail command. Using a 
simple configuration file, this command can reach out to 
IMAP and POP servers and download mail in batch. See 
man fetchmail. 


e Run a mail server, such as postfix, for complex mail 
delivery situations; see “Email Servers” on page 266. 


Control local mail delivery in sophisticated ways with the 
procmail command, which filters arriving email messages 
through any arbitrary program. See man procmail. 


e Perform spam filtering with the SpamAssassin suite of 
programs. Run it personally on your incoming email or at 
the server level for large numbers of users. 


In short, email is not limited to the features of your mail- 
reading program. Investigate and experiment! 


Email Servers 


Configuring a mail server is a complex job that can’t be taught 
in a few pages, so I just present some common operations. 
For Postfix, one of the most popular mail servers, I assume it’s 
already running on your local host. In case you don’t have a 
mail server yet, I also present Nullmailer, a simpler service that 
relays your mail to another server for delivery. 
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Postfix: A Full-Featured Mail Server 


Postfix is a powerful and popular mail server, and it’s controlled 
by the postfix command. Important configuration files for the 
server are located in the directory /etc/postfix: 


main.cf 
Variables that control Postfix’s behavior, such as your 
server’s name and domain, locations of important files, 
size limits on mailboxes and incoming mail, and more. 
After changing this file, run sudo postfix reload for the 
changes to take effect. 


master.cf 
Defines how Postfix runs various services. Most users 
don't commonly edit this file, but if you do, run sudo 
postfix reload for the changes to take effect. 


postfix-files 
Defines the correct permissions for all Postfix files. These 
permissions are critical for email security. Most users 
wont modify this file. To restore a Postfix installation to 
these permissions, run sudo postfix set-permissions. 


sasl_passwd 
Authentication information for connecting to a remote 
SMTP provider. After changing this file, run sudo 
postmap /etc/postfix/sasl_passwd for the changes to 
take effect. The file format is: 


[smtpserver]:port username:password 


where smtpserver is the remote SMTP server, port is its 
TCP port number (optional), and username and password 
are the authentication credentials. For example: 


[smtp.example.com]:587 smith:SEEKRIT_PASSWURD 
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WARNING 


Protect all files in /etc/postfix. Keep the files saslpasswd 
and sasl_passwd.db readable only by root (mode 0600) 
because they contain passwords. 


Postfix usually runs automatically, but you can start and stop it 
by hand: 


> sudo postfix start Run the server 

> sudo postfix status Check that the server is running 
> sudo postfix stop Halt the server gracefully 

> sudo postfix abort Kill the server ungracefully 


After a mail delivery problem, to deliver all queued mail imme- 
diately, run: 


> sudo postfix flush 


For more information on Postfix, visit postfix.org. 


Nullmailer: Simple Outgoing Email 


Postfix is complicated. A simpler solution is to forward your 
local host’s email to another mail server for delivery. (You'll 
need an account on the remote mail server.) This is called 
setting up a relay-only mail server. 


Suppose you have an account “smith” on a Linux host 
example.com, and you want to configure that host to send 
email. You also have an account “sandy” on a remote server, 
mail.example.com, that already runs Postfix for delivering mail. 
Install Nullmailer on your local host, either using your distro’s 
package manager (preferred) or from https://oreil.ly/82aMq. 
Then, as the superuser, create three local files in the direc- 
tory /etc/nullmailer: 


adminaddr 
Contains the email address to receive administrative 
emails, such as Nullmailer notices. 
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defaultdomain 
Contains the domain to set for all outgoing emails. 


remotes 
Contains the login information for the remote mail server, 
including the password in plain text, so make sure this file 
is readable only by root: 


> chmod 0600 /etc/nullmailer/remotes 
Here's what the three files look like in a typical installation: 


> cd /etc/nullmailer 

> cat adminaddr 
smith@example.com 

> cat defaultdomain 
exampLle.com 

> cat remotes 

cat: remotes: Permission denied 


> sudo cat remotes The content must be on a single line 
Mail.example.com smtp --port=587 --tls --user=sandy 
--pass=... 


Your remotes file will require different values specific to your 
remote mail server. Run man nullmailer-send to learn about 
them. Once the three files are in place, enable and start the 
Nullmailer service: 


> sudo systemctl enable nullmailer 
> sudo systemctl start nullmailer 


Send a test message: 
> echo hi | mail funkydance@another.example.com 
Check that the message was relayed to mail.example.com: 


> tail /var/log/mail Or other mail log file 
...-nullmailer-send: Starting delivery: 
host: mail.example.com 
...-nullmailer-send: From: <smith@example.com> to: 
<funkydance@another .example. com> 
...-nullmailer-send: Delivery complete 


Email Servers | 269 


Web Browsing 


lynx Text-only web browser. 
curl Access online content from the command line. 


wget Download web pages and files. 


Besides the usual web browsers such as Chrome and Firefox, 
Linux offers several ways to explore the World Wide Web via 
the command line. 


lynx stdin stdout -opt --help — --version 
lynx [options] [URL] 


Lynx is a stripped-down text-only web browser. It doesn’t dis- 
play pictures or play audio or video. All browsing is done by 
keyboard, not with a pointing device. But Lynx is incredibly 
useful when you just want a quick look at a page, when the 
network is slow, or for downloading the HTML of a website. It’s 
particularly good for checking out a suspicious URL because 
Lynx doesn’t run JavaScript and won't even accept a cookie 
without asking you first. 


> lynx https://danieljbarrett.com 


Many pages wont look right, especially if they use tables or 
frames, but usually you can find your way around a site. 


Keystroke Meaning 


? Get help. 

k List all keystrokes and their meanings. 
AG Cancel a command in progress. 

q Quit Lynx. 

Down arrow Go to the next link or form field. 

Up arrow Go to the previous link or form field. 
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Enter “Click” the current link, or finish the current form field. 
Right arrow Go forward to next page, or “click” the current link. 
Left arrow Go back to previous page. 

g Go to a URL (you'll be prompted to enter it). 

p Save, print, or mail the current page. 

Space bar Scroll down. 

b Scroll up. 

AA Go to top of page. 

AE Go to end of page. 

m Return to the main/home page. 

/ Search for text on the page. 

a Bookmark the current page. 

v View your bookmark list. 

r Delete a bookmark. 


= Display properties of the current page and link. 


\ View HTML source (press again to return to normal view). 


Lynx has over one hundred options, so the manpage is worth 
reading. Other text-based browsers include w3m, links, and 
elinks. 


Useful options 
-dump Print the rendered page to standard output and exit. 
(Compare to the - source option.) 
-source Print the HTML source to standard output and exit. 


(Compare to the wget command.) 


-useragent=name_Ifasite is blocking Lynx, set the useragent to present as 
a different browser. Try -useragent=mozillLa. 
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-emacskeys Make Lynx obey emacs-like keystrokes. 
-vikeys Make Lynx obey Vim-like keystrokes. 
-homepage=URL Set your home page URL to be URL. 


-color Turn colored text mode on. 
-nocolor Turn colored text mode off. 
curl stdin stdout -file --opt --help  --version 


curl [options | URLs] 


The curl command accesses online content. It can download 
web pages, test a web service, or do any other task that hits a 
URL. By default, curl writes to standard output, so redirect the 
output as needed or use the -o option: 


> curl https://www.yahoo.com > mypage.html Redirect 
> curl -o mypage.html https: //www. yahoo.com -o option 
> curl https://www.yahoo.com | we -l Pipelines 


curl is great for quickly testing a REST API. Search Wikipedia 
for the word “Linux” and receive a JSON response: 


> curl "https: //en.wikipedia.org/w/rest.php/vi/search/\ 
page?q=Linux" 
{"pages":[{"id":6097297,"key":"Linux","title": "Linux" ... 


Curl has dozens of options and can speak numerous protocols 
besides HTTP, such as IMAP, FTP, and SMB. It can add web 
headers, post data to web pages, authenticate with usernames 
and passwords, handle SSL certificates, simulate cookies, and 
much more (see the manpage). 


Useful options 


-o file Write output to the given file. 


-A name Set the user agent to name. Try -A mozilla. 
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-V Verbose mode: print lots of diagnostic output. 


-s Silent mode: no diagnostic output. 


wget stdin stdout -file --opt --help  --version 
wget [options] URL 


The wget command downloads data from a URL to a file. It’s 
great for capturing individual web pages, downloading files, or 
duplicating entire website hierarchies to arbitrary depth. For 
example, let’s capture the Yahoo! home page: 


> wget https://www.yahoo.com 
2023-10-22 13:56:53 (3.45 MB/s) - 'index.html' saved 


which is saved to a file index.html in the current directory. wget 
can resume a download if it gets interrupted in the middle, say, 
due to a network failure: just run wget -c with the same URL 
and it picks up where it left off. 


Perhaps the most useful feature of wget is its ability to down- 
load files without needing a web browser: 


> wget https://linuxpocketguide.com/sample. pdf 


This is great for large files like videos and ISO images. You can 
even write shell scripts to download sets of files if you know 
their names. Here's a script that downloads three MP4 videos 
named 1.mp4, 2.mp4, and 3.mp4 from the root of a website. 


> for i ini 23 
do 

wget https://example.com/$i.mp4 
done 


wget has over 70 options, so I cover just a few important ones. 
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Useful options 


-U name 


-O filename 
-i filename 


=¢ 


-t N 
--progress=dot 
--spider 


-nd 


el 


-A pattern 
-R pattern 
-I pattern 


-X pattern 


Ifa site is blocking wget, set the useragent to present as a 
different browser. Try -U mozilla. 


Write all the captured HTML to the given file. 
Read URLs from the given file and retrieve them in turn. 


Continue an interrupted download. For example, if wget 
downloaded 100K of a 150K file, wget -c retrieves just 
the remaining 50K. wget can be fooled, however, if the 
remote file has changed since the first (partial) download, 
50 use -c only if the remote file hasn't changed. 


Try N times before giving up. N=0 means try forever. 
Print dots instead of bars to show download progress. 
Don’t download, just check existence of remote pages. 


Retrieve all files into the current directory, instead of 
duplicating the remote directory hierarchy. 


Retrieve a whole tree of pages recursively. 
Retrieve files at most N levels deep (5 by default). 


Inside retrieved files, modify URLs so the files can be 
viewed locally in a web browser. 


Download all necessary files to make a page display 
completely, such as stylesheets and images. 


Follow relative links (within a page) but not absolute links. 


Accept mode: download only files whose names match a 
given pattern, using standard shell pattern-matching. 


Reject mode: download only files whose names do not 
match a given pattern. 


Directory inclusion: download files only from directories 
that match a given pattern. 


Directory exclusion: download files only from directories 
that do not match a given pattern. 
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CHAPTER 6 


Getting Stuff Done 


Screen Output 


echo Print simple text on standard output. 

printf Print formatted text on standard output. 

yes Print repeated text on standard output. 

seq Print a sequence of numbers on standard output. 

clear — Clear the screen or window. 
Linux provides several commands to print messages on stan- 
dard output. Each has different strengths and intended pur- 
poses. These commands are invaluable for learning about 


Linux, debugging, writing shell scripts (see “Programming with 
Shell Scripts” on page 311), or just talking to yourself. 


echo stdout 
echo [options] strings 
The echo command simply prints its arguments: 


> echo We are having fun 
We are having fun 
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Confusingly, there are several different echo commands with 
slightly different behavior. There’s /bin/echo, but Linux shells 
typically override this with a built-in command called echo. To 
find out which you're using, run the following command: 


> type echo 
echo is a shell builtin 


Useful options 


-n 


=e 


-E 


Don’t print a final newline character. 


Recognize and interpret escape characters. For example, try echo 
"hello\a' andecho -e 'hello\a’. The first prints literally and 
the second produces a beep. 


Don't interpret escape characters: the opposite of -e. 


Available escape characters are: 


\a 
\b 
\c 
\f 
\n 
\r 
\t 
\v 
\\ 
\! 
\" 


Alert (play a beep) 

Backspace 

Don't print the final newline (same effect as -n) 
Form feed 

Line feed (newline) 

Carriage return 

Horizontal tab 

Vertical tab 

A backslash 

Single quote 


Double quote 


\nnn_ The character whose ASCII value is nnn in octal (base 8) 
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printf stdout --opt --help 
printf format_string [arguments] 


The printf command is an enhanced echo: it prints formatted 
strings on standard output. It operates much like the C pro- 
gramming language function printf(), which applies a format 
string to a sequence of arguments to create some specified 
output. For example: 


> printf "User %s is %d years old.\n" sandy 29 
User sandy is 29 years old. 


The first argument is the format string, which in our example 
contains two format specifications, %s and %d. The subsequent 
arguments, sandy and 29, are substituted by printf into the 
format string and then printed. Format specifications can get 
fancy with floating-point numbers: 


> printf "That'll be $%0.2f, my friend.\n" 3 
That'll be $3.00, my friend. 


Linux has two printf commands: one built into bash, and one 
in /usr/bin/printf. The two behave almost identically with small 
differences; for example, only the built-in printf can store 
output in a shell variable (with the -v option). 


Make sure the number of format specifications (%) equals the 
number of arguments to printf. With too few arguments, 
printf outputs default values (0 for numeric formats, an empty 
string for string formats). With too many arguments, printf 
iterates over the extras until they run out. Treat such mis- 
matches as errors, even though printf is forgiving: they are 
bugs waiting to happen. 


> printf "%d %d\n" 10 Too few arguments 
10 0 

> printf "%d %d\n" 10 20 30 40 Too many arguments 
10 20 

30 40 
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Format specifications are described in detail on the manpage 
for the C function printf (see man 3 printf). Here are some 
useful ones: 

%d Decimal integer 

%ld Long decimal integer 

%o — Octal integer 

%x Hexadecimal integer 

%f — Floating point 

%Lf Double-precision floating point 

%c — Asingle character 

%s String 

%q String with any special shell characters escaped 

%%  Apercent sign by itself 
Set the minimum width of the output by inserting a numeric 
expression just after the leading percent sign. For example, 
“%5d” means to print a decimal number in a five-character- 
wide field, and “%6.2f” means a floating-point number in a 
six-character-wide field with two digits after the decimal point. 
Some useful numeric expressions are: 

n Minimum width n 

On Minimum width n, padded with leading zeros 

nm Minimum width n, with m digits after the decimal point 
printf also interprets escape characters like “\n” (print a new- 


line character) and “\a” (ring the bell). See the echo command 
for the full list. 
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yes stdout -help — --version 


yes [string] 


«9 


The yes command prints “y” or a given string forever, one 


string per line: 
> 
y 
y 
sd 


Though it might seem useless at first glance, yes can be perfect 
for commands that prompt the user to continue. Pipe the out- 
put of yes into the input of the command to answer every 
prompt affirmatively: 


> yes | any_interactive_command 


When any_interactive_command terminates, so does yes. 


seq stdout --help = --version 
seq [options] specification 


The seq command prints a sequence of integers or real num- 
bers that’s suitable to pipe to other programs. It supports three 
kinds of arguments: 


A single number: an upper limit 
seq begins at 1 and counts up to the number: 


> seq 3 
1 
2 
3 
Two numbers: lower and upper limit 


seq begins at the first number and counts as far as it can 
without passing the second number: 
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seq 2 5 


= 
2 
3 
4 
5 


Three numbers: lower limit, increment, and upper limit 
seq begins at the first number and increments by the sec- 
ond number as far as possible without passing the third 
number: 


seq 1 .3 2 


> 
1 

1. 
1. 
1. 


OoOnwW 


You can also go backward with a negative increment: 


seq 5 -12 


- 
5 
4 
3 
Z 


Useful options 


-W Print leading zeros, as necessary, to give all lines the same width: 
> seq -w 8 10 
08 
09 
10 


-f format Format the output lines with a printf-like format string, which 
must include either %g (the default), %e, or %F: 
> seq -f '**%g**' 3 
KRY RK 
RRDEK 


KRBRK 
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-s string Use the given string as a separator between the numbers. By 
default, a newline is printed (i.¢., one number per line): 
> seq -s ':' 10 
2:2:324:526:7:8:9:10 


Bash and other shells have a similar feature to produce a 
sequence of numbers; see “Brace Expansion” on page 39. 


> echo {1..10} 
123456789 10 


clear stdout 


clear 


This command simply clears your display or shell window. 
Alternatively, press *L. 


Copy and Paste 


xclip Copy and paste between the shell and the clipboard. 
xsel Manipulate the clipboard from the shell. 


Linux has a clipboard to copy and paste between graphical 
applications. Actually, Linux has three different clipboards, 
called selections. You can access the selections from the com- 
mand line, send the output of any shell command to the selec- 
tion, or read the selection like standard input. 


These commands work only if your shell runs in an X11 
environment such as GNOME or KDE. Wayland has other 
clipboard mechanisms such as wl-copy and wl-paste. In a 
non-graphical environment, the tmux and screen commands 
provide clipboards while running. If you're using none of these 
environments, no clipboard exists. 
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xclip stdin stdout -help — --version 
xclip [options] 


xclip reads and writes the three Linux selections (clipboards) 
to copy and paste text between the shell and graphical applica- 
tions. To see it in action, copy some text with your mouse to a 
selection—say, double-click a word in your terminal window— 
and then run: 


> xclip -o 


The text you copied is printed on standard output. As another 
example, copy the contents of a file to a selection, and then 
print the selection: 


> cat poem See the file 

Once upon a time, there was 

a little operating system named 

Linux, which everybody loved. 

> cat poem | xclip -t Pipe file to selection 
> xclip -o Print selection 
Once upon a time, there was 

a little operating system named 

Linux, which everybody loved. 


All command-line options for xclip use single dashes, even 
-help and -version. 


Useful options 


-selection name Choose aselection: primary (default), secondary, 
or clipboard. In my terminal windows, the middle 
mouse button pastes from primary, and “Paste” in the 
right-button menu uses clipboard. 


-i Read the selection contents from standard input (default 
behavior). 
-0 Write the selection contents to standard output. 
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xsel stdin stdout ~-help — --version 
xsel [options] 


xsel is a more powerful version of xclip. Along with reading 
and writing the three selections (clipboards), it can also append 
to them, swap them, and clear them: 


> echo Hello | xsel -i 

> xsel -o 

Hello 

> echo World | xsel -a Append 
> xsel -o 

Hello 

World 


Useful options 


-p Use the primary selection (default). 

-s Use the secondary selection. 

-b Use the clipboard selection. 

-i Read the selection contents from standard input (default behavior). 
-a Append to the selection. 

-o Write the selection contents to standard output. 

-c Clear the selection contents. 


-x Swap (exchange) the contents of the primary and secondary selection. 


Math and Calculations 


expr Do simple math on the command line. 
be Text-based calculator. 
dc Text-based RPN calculator. 


Need a calculator? Linux provides some commands to compute 
mathematical truths for you. 
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expr stdout -help — --version 
expr expression 


The expr command does simple math (and other expression 
evaluation) on the command line: 


> expr 7 + 3 

10 

> expr '(' 7 +3 ')' '*' 14 Quote any shell characters 
140 

> expr Length ABCDEFG 

7 

> expr 15 '>' 16 

0 Zero means false 


Bash provides a shorthand for expr using a dollar sign and 
a double parenthesis: $((..)). Its convenient for calculations 
within a command line: 


> echo The answer is: $((7 + 3)) 
The answer is: 10 


Each argument must be separated by whitespace. Notice that 
you have to quote or escape any characters that have special 
meaning to the shell. Parentheses (escaped) may be used for 
grouping. Operators for expr include: 


Operator Numericoperation String operation 
+,-,*,/ Addition, 

subtraction, 

multiplication, and 

integer division, 

respectively 
% Remainder (mod) 
< Less than Earlier in dictionary 
<= Less than orequal —_ Earlier in dictionary, or 

equal 

> Greater than Later in dictionary 
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Operator Numericoperation String operation 


>= Greater than or Later in dictionary, or equal 
equal 

= Equality Equality 

!= Inequality Inequality 

| Boolean “or” Boolean “or” 

& Boolean “and” Boolean “and” 

s:regexp Does the regular expression 

regexp match string s? 
substr spn Print n characters of string 


s, beginning at position p 
(the first character is p=1) 


index s chars Return the index of the 
first position in string s 
containing a character from 
string chars. Return 0 if not 
found. Same behavior as 
the C function index(). 


For Boolean expressions, the number 0 and the empty string 
are considered false; any other value is true. When returning 
Boolean results, 0 is false and 1 is true. 


bc stdin stdout --opt --help — --version 
bc [options] [files] 


bc is a text-based calculator that reads arithmetic expressions, 
one per line, and prints the results. Unlike most other calcula- 
tors, bc can handle numbers of unlimited size and any number 
of decimal places: 


> be 
142434445 Add five numbers 
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15 


scale=2 Set precision to 2 decimal places 
(1+2%*3/4)-5 

-2.50 

24100 Raise 2 to the power 100 
1267650600228229401496703205376 

4D Exit 


Programmers may enjoy the ability to switch bases to perform 
calculations and conversions in binary, octal, hexadecimal, or 
even custom bases: 


> be 

obase=2 Display results in base 2 
999 

1111100111 

obase=16 Or base 16 

999 

3E7 


But bc doesn’t stop there. It’s also a programmable calculator 
that can define functions. Here’s a function that implements the 
quadratic formula from algebra and prints the real roots of a 
given equation, stored in a file called quadratic.txt:! 


> cat quadratic.txt 

scale=2 

define quadform ( a, b, c ) { 
rooti = (-b + sqrt(b*2 - 4*a*c)) / (2*a) 
root2 = (-b - sqrt(b*2 - 4*a*c)) / (2*a) 
print root1, " ", root2, "\n" 


} 


quadform(1, 7, 12) Solve x? + 7x + 12 = 0 
Redirect the file to bc to run the function and see the results: 


> be < quadratic.txt 
-3.00 -4.00 


1 This demonstration code fails if the roots are imaginary. 
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In its most powerful form, bc is a programming language 
for arithmetic. You can assign values to variables, manipulate 
arrays, execute conditional expressions and loops, and even 
write scripts that prompt you for values and run any sequence 
of math operations. For full details, see the manpage. 


Useful arithmetic operations 

+, -, *, / Addition, subtraction, multiplication, and division, respectively. 
Results of division are truncated to the current scale (see below). 

% Remainder (mod). 
7 Exponentiation, as in 105 for “ten to the fifth power.” 
sqrt(N) Square root of NV. 
ibase=N Treat all input numbers as base NV. 
obase=N Output all numbers in base W. 
scale=N Set the number of significant digits after the decimal point to WV. 
Cane) Parentheses for grouping (changing precedence). 


name=value Assign a value to the variable name. 


dc stdin stdout -file --opt --help  --version 
dc [options] [files] 


The dc (desk calculator) command is a reverse Polish notation 
(RPN), stack-based calculator that reads expressions from stan- 
dard input and writes results to standard output. If you know 
how to use a Hewlett-Packard RPN calculator, dc is pretty easy 
once you understand its syntax. If you're accustomed to tradi- 
tional calculators, however, dc may seem inscrutable. I cover 
only some basic commands. 


For stack and calculator operations: 
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Quit de. 
f Print the entire stack. 
c Delete (clear) the entire stack. 
p Print the topmost value on the stack. 
P Pop (remove) the topmost value from the stack. 
nk Set precision of future operations to be n decimal places (default is Ok, 


meaning integer operations). 


To pop the top two values from the stack, perform a requested 
operation, and push the result: 


+, -, *, / Addition, subtraction, multiplication, and division, respectively. 


% Remainder (mod). 
a Exponentiation (second-to-top value is the base, top value is the 
exponent). 


To pop the top value from the stack, perform a requested 
operation, and push the result: 


v_ Square root. 


Examples: 


> de 

45+ p Print the sum of 4 and 5 

9 

23%p Raise 2 to the 3rd power and print the result 

8 

10 * p Multiply the stack top by 10 and print the result 
80 

f Print the stack 


+p Pop the top two values and print their sum 


4D Exit 
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Dates and Times 


cal  Printa calendar. 


date Print or set the date and time. 


Need a date? How about a good time? Try these commands to 
display and set dates and times on your system. 


cal stdout 


cal [options] [month [year]] 


The cal command prints a calendar—by default, the current 
month: 


> cal 
January 2024 
Su Mo Tu We Th Fr Sa 
423 4 5 6 
7 $8 9 10 11 12 13 
14 15 16 17 18 19 20 
21 22 23 24 25 26 27 
28 29 30 31 


To print a different calendar, supply a month and four-digit 
year: cal 8 2024 or cal aug 2024. If you omit the month (cal 
2024), the entire year is printed. Note that several different cal 
programs exist and yours may behave differently. 


Useful options 


-y Print the current year’s calendar. 
-3 Three-month view: print the previous and next month as well. 


-j Print each date by its position in the year. February 1 would be displayed as 
32, February 2 as 33, and so on. 
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date stdout ~-help — --version 
date [options] [format] 


The date command prints dates and times. The results depend 
on your system's locale settings (for your country and lan- 
guage). In this section, I assume an English, US-based locale. 


By default, date prints the system date and time in the local 
timezone: 


> date 
Sun Jun 4 02:09:05 PM EDT 2023 


Format the output differently by supplying a format string 
beginning with a plus sign: 


> date +%x 

06/04/2023 

> date '+The time is %1L:%M %p on a lovely %A in %B' 
The time is 2:09 PM on a lovely Sunday in June 


Here is a sampling of the date command's many formats: 


Format Meaning Example (US 
English) 

Whole dates and times: 

%C Full date and time, 12-hour clock Sun 04 Jun 2023 
02:09:05 PM EDT 

%D Numeric date, 2-digit year 06/04/23 

%x Numeric date, 4-digit year 06/04/2023 

%T Time, 24-hour clock 14:09:05 

%X Time, 12-hour clock 02:09:05 PM 

Words: 

%a Day of week (abbreviated) Sun 

%A Day of week (complete) Sunday 

%b Month name (abbreviated) Jun 


290 | Chapter 6: Getting Stuff Done 


Format Meaning Example (US 


English) 
%B Month name (complete) June 
%Z Time zone EDT 
%p AM or PM PM 
Numbers: 
Ww Day of week (0-6, 0=Sunday) 0 
%u Day of week (1-7, 1=Monday) 7 
%d Day of month, leading zero 04 
%e Day of month, leading blank 4 
%j Day of year, leading zeros 144 
om Month number, leading zero 06 
%y Year, 2 digits 23 
%Y Year, 4 digits 2023 
%M Minute, leading zero 09 
%S Seconds, leading zero 05 
%L Hour, 12-hour clock, leading blank 2 
%I Hour, 12-hour clock, leading zero 02 
%k Hour, 24-hour clock, leading blank 14 
%H Hour, 24-hour clock, leading zero 14 
%N Nanoseconds 384789400 
%S Seconds since the beginning of Linux 1685902145 
time: midnight January 1, 1970 
Other: 
%n Newline character 
%t Tab character 
6% Percent sign % 


DatesandTimes | 291 


date also displays other dates and times via options. 


Useful options 


-d string _ Display the given date or time string, formatted as you wish. 


-r filename Display the last-modified timestamp of the given file, 
formatted as you wish. 


-s string  Setthe system date and/or time to be string; only the 
superuser can do this. 


Version Control 


git Perform version control using Git. 


svn_ Perform version control using Subversion. 


Version control systems like Git and Subversion let you keep 
track of changes to files, safely revert risky edits, and collabo- 
rate with teams to modify the same files without clobbering 
each others’ work. I'll assume you already understand version 
control concepts like repositories, branches, and commits. 


git stdout opt -help — --version 
git [options] subcommand [arguments] 


Git is a popular and powerful version control system. It gives 
every participant their own full-featured repository and lets 
them share changes with other repositories. The git command 
has a rich set of features too large to teach here, but let’s check 
out some common operations. 


To get started, either create a local repository: 


> mkdir project 
> cd project 
> git init . 
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or download a copy of a remote repository by its URL: 
> git clone url 


After you edit some files, a common sequence of git com- 
mands is as follows: 


> git status List the changed files 

> git diff Check changes line by line 

> git add -A Stage the changed files 

> git diff --staged Check the staged changes 

> git commit -m"comment" Commit the staged changes locally 
> git show View the committed changes 


Staging and Undoing Changes 


Staging means copying files to a hidden “staging area” to com- 
mit in the next git commit. The opposite is git reset, which 
removes (“unstages”) files from the staging area. 


> git reset files... Unstage some files 
> git reset Unstage ALL files 


You can also undo changes that aren't staged yet, restoring some 
or all files to their most recent commit. 


> git restore files... Undo unstaged changes in files 
> git reset --hard Unstage, then undo ALL changes 


Older versions of Git use git checkout instead of git restore. 


WARNING 


git reset --hard destroys all uncommitted changes, 
whether they’re staged or not. Be sure you mean it! 
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If youre working with a remote repository, push your changes 
there at your convenience. The following git command pushes 
your changes to the remote branch that matches your local one. 


> git push origin HEAD 


Pull your teammates’ changes into your local repository. This 
is a good practice to ensure your changes work with your 
teammates’ before you push. 


> git pull 


Another common workflow is to create a branch, commit var- 
ious changes to it, merge the changes into the main branch 
when youre ready, and delete the branch when you're done.” 


> git switch -c branch_name Create local branch 

ae Edit files and commit changes 
> git switch main Switch to your main branch 
> git merge branch_name Merge your changes 

> git branch -d branch_name Delete local branch 


You can also push changes to a remote branch other than the 
main one: 


> git push origin branch Push changes 


and delete a remote branch when your collaborators are fin- 
ished with it: 


> git push origin --delete branch Delete remote branch 
View your branch's change history: 
> git log 


Each commit has a revision ID: a string of 40 hexadecimal 
digits, sometimes abbreviated to the first 7 digits. Many git 


2 Older Git versions use git checkout [-b] instead of git switch 


[-c]. 
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commands accept revision IDs as arguments, like git log 
b77eb6b. 


Other common operations: 


> git branch List all branches 

> git mv source dest Move a file/directory 

> git cp source dest Copy a file/directory 

> git rm file Delete a file 

> git rm -r dir Delete a directory 

svn stdout -opt --help — --version 


svn subcommand [options] [arguments] 


Subversion is a version control system for a tree of files. Unlike 
Git, Subversion requires an existing repository before you can 
work on files, so I assume you already have access to a reposi- 
tory. The svn command has a rich set of features too large to 
teach here, but let’s check out some common operations. 


To get started, locate the URL of an existing repository and 
check it out, creating a local workspace called the working copy: 


> svn checkout URL 


After you edit files in the working copy, a common sequence of 
svn commands is as follows: 


> svn status List the changed files 
> svn diff Review your changes 
> svn commit -m"comment" Commit your changes 


If you create new files, you must tell Subversion that they're 
part of your working copy before committing them: 


> svn add new_files... Add to working copy 
> svn commit -m"comment" Commit your changes 
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Pull your teammates’ changes into your working copy. This 
is a good practice to ensure your changes work with your 
teammates’ before you commit. 


> cd root_of_your_working_copy 
> svn update 


Another common workflow is to create a branch in the reposi- 
tory, commit various changes to it, and merge the changes into 
the main branch when you're ready. Subversion’s branching 
syntax is trickier than Git’s. A branch name is typically a URL 
within the repository, like the following examples: 


https://example.com/path Remote server, SSL 
svn://example.com/path Remote server, SVN protocol 
svn://localhost/path Local server, SVN protocol 


The following commands create a branch by copying the main 
branch, often called the trunk. They then commit changes to 
the new branch and merge it back into the trunk. 


> ls 

my_project 

> svn cp my_project my_branch Create branch 

> nano my_branch/file. py Edit a file 

> svn commit -m"edited in a branch" Commit to branch 
Adding my_branch 

Sending my_branch/file. py 


Transmitting file data .done 

Committing transaction... 

Committed revision 966. 

> cd my_project 

> svn merge -c966 svn://example.com/my_branch . Merge 
--- Merging r966 into .: 


U file.py 
> svn commit -m"merging into trunk" Commit to trunk 
Sending file.py 


Transmitting file data .done 
Committing transaction... 
Committed revision 967. 
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View your change history: 
> svn log 


Each commit has a revision ID: a positive integer that increa- 
ses by 1 with each commit. IDs appear in the log. Many svn 
commands accept revision IDs as options, like svn log -r25. 
Specify a range of revisions with two IDs separated by a colon: 
svn log -r25:28. 


Other operations: 


> svn mv source dest Move a file/directory 
> svn rm path Delete a file/directory 
> svn cat [-r revision] file Print file on stdout 
> svn info path Print info about files 
Containers 


docker Package and run applications in containers. 


Have you ever installed an application and discovered that your 
Linux system isn’t configured to run it? Containers are a way 
to package up an application’s code plus all its dependencies so 
its ready to run on other systems. Even better, each running 
container is an isolated, miniature environment, sort of like 
a virtual machine but much lighter weight, which is great for 
scaling up applications, especially in the cloud. 


Docker is a popular infrastructure for containers. I present its 
common uses and assume you have a Docker environment. 


docker stdout --help —--version 
docker subcommand [options] [arguments] 


The docker command creates, runs, and manages containers. 
Begin with a text file named Dockerfile that specifies a running 
container’s contents and behavior. Here’s a simple Dockerfile for 
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an image that simply sleeps for five minutes, but it could just as 
easily run a web server or do calculations. 


# A image that sleeps for 5 minutes. 

# Based on a minimal image called alpine. 
FROM alpine 

CMD ["sleep", "300"] 


Then “build” the Dockerfile to create an image that’s ready 
to run. Finally, run one or more instances of the image, each 
inside a container that is an isolated (sandboxed) process. A 
typical workflow is: 


> docker build -t example . Build an image named “example” 


> docker run example & Run image in a container 

> docker run example & Run image in another container 
> docker ps List running containers 
CONTAINER ID IMAGE COMMAND 


ff6d0f5ad309 example "sleep 300" 
80519480635 example "sleep 300" 


Or, instead of creating a Dockerfile, download and run an 
image created by others. The Docker Registry is an infrastruc- 
ture for sharing images. A typical workflow is: 


> docker search hello Search the Registry 
NAME DESCRIPTION 
hello-world Hello World! (an example of minimal... 


> docker pull hello-world Download an image 
> docker run hello-world = Run image in a container 


NOTE 


Depending on how docker is installed on your system, you 
may need to run docker commands as root (sudo docker) 
or join the Linux group “docker” to run commands as 
yourself (sudo usermod -aG docker SUSER) . 
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Common Docker Operations 


For operations that need a container name or ID, run docker 
ps to find them. Most docker subcommands have options, and 
there are other subcommands I don't cover. The Docker site 
has full details. 


Images and Containers 


docker 


docker 
docker 


docker 


search string 


pull image 
build image 


create image 


Executing Containers 


docker 


docker 


docker 


docker 


docker 


docker 


docker 


run image 


stop names 


start names 


restart names 


Pause names 


unpause names 


kill names 


Locate an image in a central registry with a name 
or description matching string. 


Download an image from a central registry. 
Given a Dockerflle, build an image. 


Given an image, create a container and don’t run 
it. 


Create a container and run an instance of an 
image. 


Stop the execution of containers with the given 
names or IDs. For a time, the containers can be 
restarted with docker start, butif they're 
stopped for long enough, they are killed. 


Start any stopped containers with the given 
names or IDs. 


Same as docker stop followed by docker 
start. 


Suspend all processes in containers with the 
given names or IDs. The container itself continues 
to run. 


Unpause containers with the given names or IDs. 


Kill containers with the given names or IDs. 
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Manipulating Containers and Images 


docker cp path1 path2 Copy files to and from a container. One path is 
in your local filesystem and the other is within 
the container. Paths within a container begin 
with the container name or ID, a colon, and 
then a file path, like 45771¢25a5a0;/etc. 


docker exec name cmd _— Execute a command inside a container, such as 
docker exec dcf10812030b ls -l. 


docker diff name List file additions, removals, and changes in a 
container since it was run. 


docker rename old new Rename a container. 


docker rm [-f] name Remove (delete) a container. Use - f to force a 
deletion. 


docker rmi [-f] image Remove (delete) an image. Use - f to force a 


deletion. 
Monitoring Containers 
docker ps [-a] List all running containers. Add - a to 
list all containers. 
docker top name List processes running in the given 
container. 


docker logs [options] name View the system log of a given 
container. The option - - fol Low prints 
the logs and continues monitoring 
them. The option --tail prints the 
last N log lines. The option --since 7 
begins the logs at timestamp 7, such as 
--since 2024-01-31. 


I’ve covered just the basics of the docker command. See the 
manual to learn more. 
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Displaying and Processing Images 


display Display a graphics file. 

convert Convert files from one graphical format into another. 

mogrify Modify a graphics file. 

montage Combine graphics files. 

For viewing or editing graphics, Linux has handy tools with 
tons of options. Let’s focus on command-line tools from a 


package called ImageMagick. Its commands all have similar 
usage, and a full explanation is at https://oreil.ly/nJZnm. 


display stdin -file --opt --help — --version 
display [options] files 


The display command displays images of numerous formats: 
JPEG, PNG, GIF, BMP, and more. It also includes a small suite 
of image editing tools that appear if you left-click the displayed 
image. Type q to exit the program. 


> display photo. jpg 
The command is very powerful, with more than 100 options 


listed on its manpage. 


Useful options 


-resize size Resize the image. The size values are extremely flexible, 
including setting the width (800), the height (x600), both 
(800x606), a percentage to grow or shrink (50%), an area in 
pixels (480000@), and more. 


-flip Reverse the image vertically. 
-flop Reverse the image horizontally. 
-rotate N Rotate the image by N degrees, positive or negative. 
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-backdrop Display the image on a backdrop of solid color that covers the 


rest of your screen. 
-fill Set the solid color used by the - backdrop option. 
-delay N Show the image for NV seconds and then exit. If you list 


multiple images, you get a slideshow with a delay of N 
seconds between images. 


-identify Print information about the image’s format, size, and other 
statistics to standard output. 


convert stdin stdout -file --opt --help  --version 
convert [input_options] infile [output_options] outfile 


The convert command copies an image and converts it to a 
different graphics format. For example, if you have a JPEG file, 
produce a PNG or PDF file of the same image: 


> convert photo.jpg newphoto.png 
> convert photo.jpg newphoto. pdf 


Modify the copy at the same time, such as resizing or reversing 
it: 


> convert photo.jpg -resize 50% -flip newphoto.png 


convert accepts largely the same options as display. 


mogrify stdin stdout -file --opt --help  --version 
mogrify [options] file 


The mogrify command transforms an image like convert does, 
but it changes the original image file you provide, not a copy. 
(So convert is a safer command when experimenting on a 
favorite photo.) It accepts largely the same options as display: 


> mogrify -resize 25% photo. jpg 
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montage stdin stdout -file --opt --help  --version 
montage infiles [options] outfile 


montage produces a single image file from a collection of input 
files. For example, create a sheet of thumbnails as a single 
image, labeling each thumbnail with its original filename: 


> montage photo.jpg photo2.png photo3.gif \ 
-geometry 120x176+10+10 -Label '%f' outfile.jpg 


montage provides great control over how those images appear. 
The preceding command, for example, produces thumbnails 
of size 120 x 176 pixels, offset by 10 pixels horizontally and 
vertically (creating space between the thumbnails), and labeled 
with their input filename. 


Useful options 


-geometry WxH[+|-]X[+|-]Y Set the width (W), height (H), and (X,Y) 
offset of the images. Example value: 
120x176+10-10. 


-frame N Draw a frame of N pixels around each 
image. 
-label string Label each image with any string, which 


can contain special escape characters 
beginning with a percent sign: %f for the 
original filename, %h and %w for height 
and width, %m for file format, and about 
40 others. 


Audio and Video 


mediainfo Print details about a multimedia file. 
cdparanoia Rip audio from CDs to WAV files. 
lame Convert from WAV to MP3. 
id3info View ID3 tags in an MP3 file. 
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id3tag Edit ID3 tags in an MP3 file. 
ogginfo View information about an 0GG file. 


metaflac View and edit information about a FLAC file. 


SOx Convert between audio file formats. 
mplayer Play a video or audio file. 
ffmpeg Convert between video and/or audio file formats. 


There are numerous Linux programs with graphical interfaces 
for playing and editing audio and video, but let’s focus once 
again on command-line tools. 


mediainfo stdout - file --help — --version 
mediainfo [options] files 


The mediainfo command displays details about video and 
audio files. 


> mediainfo clip.mp4 


General 

Complete name : clip.mp4 

Format : MPEG-4 

Format profile : Base Media / Version 2 
Codec ID : mp42 (mp42/mp41/isom/avc1) 
File size : 1.11 MiB 

Duration : 23s 275 ms 


Overall bit rate : 399 kb/s 


mediainfo has options, but usually the default output is suffi- 
cient. If you need more, add the option -f (“full”) to print every 
picky detail about a media file. 
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cdparanoia --opt --help — --version 
cdparanoia [options] span [outfile] 


The cdparanoia command reads (rips) audio data from a CD 
and stores it in WAV files (or other formats: see the manpage). 
Common uses are: 


cdparanoia N 
Rip track N to a file. 


cdparanoia -B 
Rip all tracks on the CD into separate files. 


cdparanoia -B 2-4 
Rip tracks 2, 3, and 4 into separate files. 


cdparanoia 2-4 
Rip tracks 2, 3, and 4 into a single file. 


If you experience difficulty accessing your drive, try running 
cdparanoia -Qvs (meaning “search for CD-ROM drives ver- 
bosely”) and look for clues. 


lame -file --help —--version 
lame [options] file.wav 


The lame command converts a WAV audio file (say, song. wav) 
into an MP3 file: 


> Lame song.wav song2.mp3 


It has over one hundred options to control bit rate, convert 
other formats, add ID3 tags, and much more. 
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id3info --help —--version 
id3info [options] [files] 


The id3info command displays information about an MP3 
audio file, such as the song title, recording artist, album name, 
and year. The file must contain ID3 tags. There are no options 
except --help and --version: 


> id3info knots.mp3 

*** Tag information for knots.mp3 

=== TIT2 (Title/songname/content description): Knots 
=== TPE1 (Lead performer(s)/Soloist(s)): Gentle Giant 
=== TALB (Album/Movie/Show title): Octopus 

=== TYER (Year): 1972 

*** mp3 info 

MPEG1/layer III 

Bitrate: 320KBps 

Frequency: 44KHz 


id3tag -file --opt --help  --version 
id3tag [options] files 


The id3tag command adds or modifies ID3 tags in an MP3 file. 
For example, to tag an MP3 file with a new title and artist, run: 


> id3tag -A "My Album" -a "Loud Linux Squad" song.mp3 
Useful options 


-A name _ Set the artist's name. 
-a title  Setthealbumtitle. 
-s title  Setthesong title. 

-y year _ Setthe year. 

-t number Set the track number. 


-g number Set the genre number. 
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ogginfo ~ opt 
ogginfo [options] [files] 


ogginfo is a simple command that displays information about 
an Ogg Vorbis audio file: 


> ogginfo knots.ogg 
Processing file "knots.ogg"... 


User comments section follows... 
Title=Knots 
Artist=Gentle Giant 
Album=Octopus 


Vorbis stream 1: 
Total data length: 69665 bytes 
Playback length: 0m:05.067s 
Average bitrate: 109.973744 


Add the -h option for more detailed usage information. 


metaflac --opt —help — --version 
metaflac [options] [files] 


The metaflac command displays or changes information about 
a FLAC audio file. To display information, run: 


> metaflac --list knots.flac 


sample_rate: 44100 Hz 
channels: 2 
bits-per-sample: 16 
total samples: 223488 


comments: 4 
comment[0]: Title=Knots 
comment[1]: Artist=Gentle Giant 
comment[2]: Album=Octopus 
comment[3]: Year=1972 
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The simplest way to change information, such as the title and 
artist, is to export the information to a text file, edit the file, and 
then reimport it: 


> metaflac --export-tags-to info.txt knots.flac 
> cat info.txt 

Title=Knots 

Artist=Gentle Giant 

Album=Octopus 

Year=1972 

> nano info.txt Make changes and save the file 

> metaflac --import-tags-from info.txt knots. flac 


Useful options 


--show-tag name _ Display the value for the named tag, such as 
title, artist, album, year, etc. There are 
many other “show” options for other information: 
see the manpage. 


--remove-tag name Remove all occurrences of the named tag (title, 
artist, etc.) from the FLAC file. 


SOx -file --opt --help — --version 
sox [options] infile outfile 


sox is the simplest command to convert from one audio file 
format to another. It supports MP3, OGG, FLAC, WAV, and 
dozens of other formats. (Run man soxformat for a list.) Simply 
specify the new format using the correct file extension, as in 
these examples: 


> sox knots.mp3 knots2.wav MP3 to WAV 
> sox knots.ogg knots2.mp3 OGG to MP3 
> sox knots.flac knots2.0gg FLAC to OGG 


sox has many other uses, including combining audio files and 
adding special effects. See the manpage for details. 
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Useful options 


-S Show a progress meter; useful for long conversions. 
--no-clobber Don't overwrite the output file if it already exists. 


-t type Specify the type of the input file, if sox cannot figure it out. 
Seeman soxformat for the list of types. 


mplayer stdin -file --opt --help — --version 
mplayer [options] video_files... 


The mplayer command plays video and audio files in many 
formats (MPEG, AVI, MOV, and more): 


> mplayer clip.mp4 


Press the space bar to pause and resume the video, the cursor 
keys to jump forward and backward, and Q to quit. mplayer 
also plays audio files and has dozens of options on its manpage. 
Learn more at https://oreil.ly/96S5B. 


Other popular video players for Linux include (vlc), (kaf 
feine), and (xine). 


ffmpeg help 
ffmpeg [input_options] -i input [output_options] output 


The ffmpeg command converts video file formats, splits and 
appends videos, extracts audio tracks, creates thumbnail images 
for videos, and much more. To understand ffmpeg thoroughly, 
it helps to know digital video concepts like frame rates, demux- 
ing, and codecs; but in this pocket guide I cover just a few 
common uses. For more complicated tasks, your best bet is to 
search the web for someone else’s ffmpeg solution. 
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NOTE 


ffmpeg is sensitive to the order of its options, unlike most 
other Linux commands. Wherever you place the option -i 
filename on the command line, input file options come 
before it and output file options come after. If an input 
option sits among output options or vice versa, ffmpeg 
may fail mysteriously. 


Convert a video file myvideo.mp4 from MP4 to MOV format: 
> ffmpeg -i myvideo.mp4 myvideo.mov 
Extract 10 seconds of video (-t 10) beginning at the two 


minute mark (-s 00:02:00) and store it in a file extract.mp4: 


> ffmpeg -i myvideo.mp4 -ss 00:02:00 -t 10 \ 
-codec copy extract.mp4 


Extract audio from myvideo.mp4 into an MP3 file audio.mp3: 
> ffmpeg -i myvideo.mp4 -q:a 0 -map a audio.mp3 


Append multiple videos to create movie.mp4. Start with a text 
file that lists the paths to the source videos, then run ffmpeg: 


> cat videos.txt 

file 'videol.mp4' 

file 'video2.mp4' 

file 'video3.mp4' 

> ffmpeg -f concat -safe 0 -i videos.txt -c copy \ 
movie.mp4 


Create a thumbnail image from a video file myvideo.mp4 by 
extracting a single frame (-vframes 1) at the five-second mark 
(ss 5) into a JPEG file thumb.jpg, sized 160 x 120 pixels: 


> ffmpeg -ss 5 -i myvideo.mp4 -vcodec mjpeg \ 
-vframes 1 -an -f rawvideo -s 160x120 thumb. jpg 


ffmpeg has an extensive help system built in (see the manpage): 
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> ffmpeg -encoders List supported encoders 
> ffmpeg --help encoder=mpeg4 — Get details about one 


Programming with Shell Scripts 


Bash has a built-in programming language. You can write shell 
scripts (bash programs) to accomplish tasks that a single com- 
mand cannot. The command reset-lpg, supplied in the book’s 
examples directory, is a shell script that you can read: 


> less ~/linuxpocketguide/reset-1pg 


Like any good programming language, bash has variables, con- 
ditionals (if-then-else), loops, input and output, and more. 
Entire books and online courses have been written on shell 
scripting; I cover the bare minimum to get you started. For 
greater detail, run info bash, pick up a bash scripting book, or 
search the web for bash scripting tutorials. 


Creating and Running Shell Scripts 


To create a shell script, simply put bash commands into a file as 
you would type them. To run the script, you have three choices: 


Prepend #!/bin/bash and make the file executable 
This is the most common way to run scripts. Add the line: 


#!/bin/bash 
to the very top of the script file. It must be the first line of 
the file, left-justified. Then make the file executable: 

> chmod +x myscript 


Then run the script. For a script in the current directory, 
you will probably have to prepend “/” so the shell can 
locate the script. (The current directory is generally not in 
your search path for security reasons. You wouldn't want 
a malicious script named “ls” in the current directory to 
silently override the real ls command.) 


> ./myscript 
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Alternatively, move the script into a directory in your 
search path and run it like any other command: 


> myscript 


Pass to bash 
Run bash with the script filename as an argument. 


> bash myscript 


Run in current shell with source or a dot 
The preceding methods run your script as an independent 
entity that has no effect on your current shell. If you want 
a script to make changes to your current shell (setting 
variables, changing directory, etc.), run it with the source 
command or a single dot: 


> source myscript 
> . myscript 


Whitespace and Linebreaks 


Bash shell scripts are sensitive to whitespace and linebreaks. 
Because the “keywords” of this programming language are 
actually commands evaluated by the shell, you must separate 
arguments with whitespace. Likewise, a linebreak tells the shell 
that a command is complete, so a linebreak in the middle of a 
command can cut the command short. Follow the conventions 
I present here and you should be fine. (Also see my formatting 
advice in the sidebar “Style in Shell Scripts” on page 317.) 


To split a long command into multiple lines, end each line 
(except the last) with a single \ character, which means “contin- 
ued on next line.” Here’s a script with a long grep command: 


#!/bin/bash 
grep abcdefghijklmnopqrstuvwxyz file1 file2 file3 \ 
file4 files 


3 That's because the script runs in a separate shell (a child shell) that 
cannot alter the original shell. 
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Variables 
I described shell variables in “Shell Variables” on page 39: 


> MYVAR=6 
> echo SMYVAR 
6 


All values held in variables are strings, but if they are numeric, 
the shell treats them as numbers when appropriate: 


> NUMBER="10" 
> expr SNUMBER + 5 
15 


When you refer to a variable’s value in a shell script, it’s a good 
idea to surround it with double quotes to prevent certain run- 
time errors. An undefined variable, or a variable with spaces in 
its value, evaluates to something unexpected if not surrounded 
by quotes, causing your script to malfunction: 


> FILENAME="My Document" Space in the name 

> 1s $FILENAME Try to list it 

ls: My: No such file or directory — /s saw 2 arguments 

ls: Document: No such file or directory 

> 1s -1l "S$FILENAME" List it properly 

My Document Is saw only 1 argument 


If a variable name is evaluated adjacent to another string, sur- 
round it with curly braces to prevent unexpected behavior: 


> HAT="fedora" 

> echo "The plural of SHAT is SHATs" 

The plural of fedora is No variable “HATs” 
> echo "The plural of SHAT is ${HAT}s" 

The plural of fedora is fedoras What we wanted 
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Input and Output 


Scripts can print to standard output with the echo and printf 
commands, which I described in “Screen Output” on page 275: 


> echo "Hello world" 

Hello world 

> printf "I am %d years old\n" “expr 20 + 20° 
I am 40 years old 


Scripts can read from standard input with the read command, 
which grabs one line of input and stores it in a variable: 


> read name 

Sandy Smith <ENTER> 

> echo "I read the name $name" 
I read the name Sandy Smith 


See also “Command-Line Arguments” on page 323. 


Booleans and Exit Codes 


Before I describe conditionals and loops, you need the concept 
of a Boolean (true/false) test. To the shell, the value 0 means 
true or success, and anything else means false or failure. (Think 
of zero as “no error” and other values as error codes.) 


Additionally, every Linux command returns an integer value to 
the shell when the command exits. This value is called an exit 
code, exit value, or exit status. 


You can see this value in the special variable $?: 


> cat exittest 

My name is Sandy Smith and 

I really Like Ubuntu Linux 

> grep Smith exittest 

My name is Sandy Smith and A match was found... 
> echo $? 


0 ...S0 exit code is “success” 
> grep aardvark exittest 

> echo $? No match was found... 

1 ...50 exit code is “failure” 
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A command’s exit codes are documented on its manpage. 


The test Command 


The test command (built into the shell) evaluates simple 
Boolean expressions involving numbers and strings, and sets 
its exit status to 0 (true) or 1 (false): 


> test 10 -1t 5 


Is 10 less than 5? 


> echo $? 

1 No, it isn’t 

> test -n "hello" Does “hello” have nonzero length? 
> echo $? 

0 Yes, it does 


Here are common test arguments to check properties of inte- 
gers, strings, and files: 


File tests 

-d name 
-f name 
-L name 
-r name 
-w name 
-xX name 
-S name 
f1 -nt f2 
f1 -ot f2 
String tests 
S1 = s2 
S1 != s2 
*Z Si 


-n si 


File name is a directory. 

File name is a regular file. 

File name is a symbolic link. 

File name exists and is readable. 

File name exists and is writable. 

File name exists and is executable. 

File name exists and its size is nonzero. 
File f7 is newer than file f2. 

File f7 is older than file £2. 


String 57 equals string s2. 
String 57 does not equal string s2. 
String 57 has zero length. 


String 57 has nonzero length. 
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Numeric tests 


a -eq b Integers a and b are equal. 

a -ne b Integers a and b are not equal. 

a -gt b Integer a is greater than integer b. 

a -ge b Integer ais greater than or equal to integer b. 
a -lt b Integer ais less than integer b. 

a-le b Integer ais less than or equal to integer b. 


Combining and negating tests 


t1 -a t2 And: Both tests t7 and t2 are true. 
t1 -o t2 Or: Either test ¢7 or 2 is true. 
! your_test Negate the test (i.e., your_test is false). 


\( your_test \) Use parentheses for grouping, as in algebra. 


You can write tests in bash in three ways. The first uses the test 
command as I’ve already shown. The second is to surround a 
condition with double square brackets: 


> [[ 10 -1t 5 J] Is 10 less than 5? 

> echo $? 

1 No, it isnt 

> [[ -n "hello" ]] Does “hello” have nonzero length? 
> echo $? 

0 Yes, it does 


The third way, which is also supported by some other shells, is 
to use a single square bracket: 


> [ 10 -1t 5 J Is 10 less than 5? 

> echo $? 

1 No, it isn’t 

> [ -n "hello" J Does “hello” have nonzero length? 
> echo $? 

0 Yes, it does 


The single square bracket is an older syntax with some odd 
quirks. You must add whitespace after the left bracket and 
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before the right bracket. That’s because the left bracket is 
actually a command named “[”—it is an alias for test. You 
must therefore follow the left bracket with individual arguments 
separated by whitespace. You also must ensure that the final 
argument is a right square bracket, signifying the end of the 
test. If you mistakenly forget some whitespace: 


> [5 -lt 4] No space between 4 and ] 

bash: [: missing ']' 
then test sees the final argument is the string “4]” and com- 
plains that the final bracket is missing. 


Conditionals 


The if statement chooses between alternatives, each of which 
may have a complex test. The simplest form is the if-then 
statement: 


if command If exit status of command is 0 
then 

body 
fi 


Style in Shell Scripts 


Shell scripting keywords (if, then, fi, etc.) must be the first 
word on their line. That means the keyword must follow either 
a newline character or a semicolon (plus optional whitespace). 
Here are two other common layouts for an if statement. Other 
conditionals and loops can be styled similarly. 


if command; then Semicolon before then 
body 
FL 


if command; then body; fi Semicolons before then and fi 


Here's an example script with an if statement. Try running it 
with and without sudo and view the results. 
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> cat script-if 
#!/bin/bash 


if [ ‘whoami* = "root" ] 
then 
echo "You are the superuser" 
fi 
> ./script-if No output 


> sudo ./script-if 
[sudo] password: xXxXxxXXxx 
You are the superuser 


Next is the if-then-else statement: 


if command 
then 

body1 
else 

body2 
fi 


For example: 


> cat script-else 
#!/bin/bash 


if [ ‘whoami* = "root" ] 
then 

echo "You are the superuser" 
else 

echo "You are a mere mortal" 
fi 


> ./script-else 

You are a mere mortal 

> sudo ./script-else 
[sudo] password: xxxxXxxXxx 
You are the superuser 


Finally, there’s the form if-then-elif-else, which can have as 
many tests as you like: 


if command1 
then 

body1 
elif command2 
then 
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body2 
elif ... 


else 
bodyN 
fi 


For example: 


> cat script-elif 
#!/bin/bash 
bribe=20000 


if [ ‘whoami* = "root" ] 
then 
echo "You are the superuser" 
elif [ "SUSER" = "root" ] 
then 


echo "You might be the superuser" 
elif [ "Sbribe" -gt 10000 ] 


then 

echo "You can pay to be the superuser" 
else 

echo "You are still a mere mortal" 
fi 


> ./script-elif 
You can pay to be the superuser 


The case statement evaluates a single value and branches to an 
appropriate piece of code: 


> cat script-case 
#!/bin/bash 
echo -n "What would you like to do (eat, sleep)? " 
read answer 
case "Sanswer" in 
eat) 
echo "OK, have a hamburger." 
sleep) 
echo "Good night then." 


echo "I'm not sure what you want to do." 


Programming with Shell Scripts | 319 


echo "I guess I'll see you tomorrow." 
esac 
> ./script-case 
What would you like to do (eat, sleep)? sleep 
Good night then. 


The general form is: 


case value in 
expr1) 
body1 
expr2) 
body2 


a9 


expr) 
bodyN 

sy) 
body_of_else 


esac 


The value is any value, usually a variable value like $myvar, and 
expr1 through exprN are patterns (run the command info bash 
for details), with the final * like a final “else” Each body must 
be terminated by ; ; (as shown): 


> cat script-case2 
#!/bin/bash 
echo -n "Enter a letter: 
read letter 
case S$letter in 
X) 
echo "Sletter is an X" 


[aeiou]) 
echo "Sletter is a vowel" 
[0-9]) 
echo "Sletter is a digit, silly" 
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33 
*) 
echo "The letter 'Sletter' is not supported" 


a9 


esac 
./script-case2 
Enter a letter: e 
e is a vowel 


Loops 


The while loop repeats a set of commands as long as a condi- 
tion is true. 


while command While the exit status of command is 0 
do 

body 
done 


For example: 


> cat script-while 
#!/bin/bash 


i=0 
while [ $i -lt 3 ] 
do 
echo "Si" 
i=‘expr $i + 1° 
done 
> ./script-while 
0 
1 
2 


The until loop repeats until a condition becomes true: 


until command While the exit status of command is nonzero 
do 

body 
done 


For example: 
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> cat script-until 
#!/bin/bash 


i=0 
until [ $i -ge 3 ] 
do 
echo "Si" 
i=‘expr $i + 1° 
done 
> ./script-until 
0 
1 
2 


Watch out for infinite loops, which use while with a condition 
that always evaluates to 0 (true) or until with a condition that 
always evaluates to a nonzero value (false): 


i=1 
while [ $i -lt 10 ] 
do 
echo "forever" Oops: variable i never changes 
done Infinite loop 


Another type of loop, the for loop, iterates over values from a 
list: 


for variable in list 
do 

body 
done 


For example: 


> cat script-for 
#!/bin/bash 
for name in Tom Jane Harry 
do 

echo "Sname is my friend" 
done 
> ./script-for 
Tom is my friend 
Jane is my friend 
Harry is my friend 
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The for loop is handy for processing lists of files; for example, 
filenames with a certain extension in the current directory: 


> cat script-for2 
#!/bin/bash 
for file in *.docx 
do 
echo "Sfile is a stinky Microsoft Word file" 
done 
> ./script-for2 
letter.docx is a stinky Microsoft Word file 
shopping.docx is a stinky Microsoft Word file 


You can also generate a list of values and loop over them, 
using curly braces (“Brace Expansion” on page 39) or the seq 
command (see “Screen Output” on page 275): 


> cat script-seq 
#!/bin/bash 


for i in {1..20} Generates the integers 1-20 
do 

echo "iteration $i" 
done 


> ./script-seq 
iteration 1 
iteration 2 
iteration 3 


iteration 20 


Command-Line Arguments 


Shell scripts can accept command-line arguments and options 
just like other Linux commands. (In fact, some common Linux 
commands are scripts.) Within your shell script, refer to these 
arguments as $1, $2, $3, and so on: 


> cat script-args 
#!/bin/bash 
echo "My name is $1 and I come from $2" 


> ./script-args Johnson Wisconsin 


Programming with Shell Scripts | 323 


My name is Johnson and I come from Wisconsin 
> ./script-args Bob 
My name is Bob and I come from 


A script can test the number of arguments it received with $#: 


> cat script-args2 
#!/bin/bash 
if [ S# -1t 2 ] 
then 
echo "SQ error: you must supply two arguments" 
else 
echo "My name is $1 and I come from $2" 
fi 


The special value $0 contains the name of the script and is 
handy for usage and error messages: 


> ./script-args2 Barbara 
./script-args2 error: you must supply two arguments 


To iterate over all command-line arguments, use a for loop 
with the special variable $@, which holds all arguments: 


> cat script-args3 
#!/bin/bash 
for arg in $@ 
do 
echo "I found the argument Sarg" 
done 
> ./script-args3 One Two Three 
I found the argument One 
I found the argument Two 
I found the argument Three 


Exiting with an Exit Code 


The exit command terminates your script and passes a given 
exit code to the shell (see “Booleans and Exit Codes” on page 
314). By tradition, scripts should return 0 for success and 1 (or 
other nonzero value) on failure. If your script doesn't call exit, 
its exit code will be that of the last command the script runs. 
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> cat script-exit 
#!/bin/bash 
if [ S# -1t 2 ] 
then 
echo "SQ error: you must supply two arguments" 
exit 1 
else 
echo "My name is $1 and I come from $2" 
fi 
exit 0 
> ./script-exit Bob 
./script-exit error: you must supply two arguments 
> echo $? 
1 


Piping to bash 


Bash is not just a shell; it’s also a command, bash, that reads 
from standard input. This means you can construct commands 
as strings and send them to bash for execution: 


> echo we -1l myfile 


wc -l myfile 
> echo we -1 myfile | bash 
18 myfile 


WARNING 


Piping commands into bash is powerful but can also be 
dangerous. First, make sure you know exactly which com- 
mands you're sending to bash for execution. You don’t 
want to pipe an unexpected rm command to bash and 
delete a valuable file (or one thousand valuable files). 


If someone asks you to retrieve text from the web (say, 
with the curl command) and pipe it to bash, don’t do 
it blindly. Instead, capture the web page as a file (with 
curl or wget), examine it closely, and make an informed 
decision whether to execute it with bash. 
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This technique is incredibly useful. Suppose you want to down- 
load the files photol.jpg, photo2.jpg, through photo100.jpg from 
a website. Instead of typing 100 wget commands by hand, 
construct the commands with a loop, using seq to construct the 
list of integers from 1 to 100: 


> for i in “seq 1 100° 
do 
echo wget https://example.com/photo$i. jpg 
done 
wget https://example.com/photo1. jpg 
wget https://example.com/photo2. jpg 


wget https://example.com/photo100. jpg 


Yes, you've constructed the text of 100 commands. Now pipe 
the output to bash, which runs all 100 commands as if youd 
typed them by hand: 


> for i in “seq 1 100° 
do 

echo wget https://example.com/photo$i. jpg 
done | bash 


Here’s a more complex but practical application. Suppose you 
have a set of files you want to rename. Put the old names into a 
file oldnames, and the new names into a file newnames: 


> cat oldnames 
oldname1 
oldname2 
oldname3 
> cat newnames 
newname1 
newname2 
newname3 


Now use the commands paste and sed (“Manipulating Text in 
Files” on page 107) to place the old and new names side by side 
and prepend the word “mv” to each line, and the output is a 
sequence of “mv” commands: 
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> cat oldnames \ 
| paste -d' ' oldnames newnames \ 
| sed 's/4/mv /' 

mv oldfile1 newfile1 

mv oldfile2 newfile2 

mv oldfile3 newfile3 


Finally, pipe the output to bash, and the renaming takes place! 


> cat oldnames \ 
| paste -d' ' oldnames newnames \ 
| sed 's/4/mv /' \ 
| bash 


Beyond Shell Scripting 


Shell scripts are fine for many purposes, but Linux comes with 
much more powerful scripting languages, as well as compiled 
programming languages. Here are a few: 


Language Program Togetstarted... 
GG+ gcc,g++ man gcc 
https://oreil.ly/-kuj1 
Java javac _https://oreil.ly/-yvQR 
NET mono man mono 
https-//oreil.ly/lVenL 
Perl perl man perl 
https://oreil.ly/LKgdM 
PHP php man php 
https://oreil.ly/JPmlL 
Python python man python 
https://oreil.ly/vAunc 
Ruby ruby https://oreil. ly/ifm2L 
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Final Words 


Although I’ve covered many commands and features of Linux, 
there's so much more to learn. I hope you'll continue reading 
and exploring the capabilities of your Linux systems. 


To boost your Linux skills even more, check out my follow-up 
book, Efficient Linux at the Command Line. It goes beyond the 
basics to make you faster and more effective with Linux, with 
tons of practical tips and techniques. For more information, 
visit my website. 


O'REILLY* 


Efficient Linux 
at the Command Line 


Boost Your Command-Line Skills 


Daniel J. Barrett 
Author of Linux Pocket Guide 
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Symbols 

! (exclamation point) 
filename pattern, 38 
shell command history, 48 
test command, 100 

$ (dollar sign), variable value, 40 

$#, argument count, 324 

$((, expr command, 284 

$(, command substitution, 45 

$0, script name, 324 

$1, script argument, 323 

$2, exit code, 314 

$@, all arguments, 324 

% (percent), printf formatting, 
277-278 

& (ampersand), running back- 
ground jobs, 51 

&& (double ampersand), logical 
and, 44 

* (asterisk) 
case statement, 320 
filename pattern, 37, 38 
traceroute output, 256 

- (dash) 
filenames beginning with, xii 
standard input/output, xi, 282 


Index 


-- (double dash), end of options, 
xii 

. (dot) 
current directory, 29 
dot files, 38 
shell script execution, 312 

.. (dot dot) 
brace expansion, 39 
parent directory, 29 

/ (slash) 
directory separator, 28 
root directory, 27 
ZFS datasets, 236 

; (semicolon), combined com- 
mands, 44 

3; (two semicolons), case state- 
ment, 320 

< (less than), input redirection, 43 

<(, process substitution, 46 

= (equals sign), variable assign- 
ment, 39 

> (greater than), output redirec- 
tion, 44 

? (question mark), filename pat- 
tern, 38 

[ (left square bracket) 
alias for test command, 317 
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filename pattern, 38 
[ ] (square brackets) 
versus curly braces, 39 
expansion on command line, 
38 
grep regular expressions, 109 
test command, 316 
\ (backslash) 
escaping special characters, 47 
line continuation, xiii, 312 
4 (caret) 
control key, xiv, 80 
filename pattern, 38 
** (backquotes), 45 
(see also command substitu- 
tion) 
{ } (curly braces) 
expansion on command line, 
39 
for loops, 323 
grep regular expressions, 110 
shell variables, 313 
versus square brackets, 39 
| (pipe) operator, 26, 45, 325-327 
|| (double pipe), logical or, 44 
~ (tilde), home directories, 30 


A 

abiword command, 78 

absolute path, 28, 69 

absolute permissions, 92 

addgroup command, 193 

adduser command, 189 

adminaddr file, 268 

administrator (see superuser) 

alias command, 42, 62 

alphabetical order, sorting text in, 
118 

ampersand (&), running back- 
ground jobs, 51 

apt command, 196, 200 

aptitude command, 196, 201 

Arch Linux, 60, 204 


arguments for commands, 24, 277 
aspell command, 159 
asterisk (*) 
case statement, 320 
filename pattern, 37, 38 
traceroute output, 256 
at command, 176-178 
atq command, 178 
atrm command, 178 
attributes of files 
changing, 96 
listing, 84-86, 95-96 
audio, 304-310 
compressing, 305 
converting, 308 
playing, 309 
ripping, 305 
tags, 306-307 
awk command, 117, 125 


B 
background jobs, running, 50 
backquotes (*"), 45 
(see also command substitu- 
tion) 
backslash (\) 
escaping special characters, 47 
line continuation, xiii, 312 
backups, 241-247 
basename command, 69 
bash shell, 20, 36-59 
aliases, 42 
brace expansion, 39 
built-in commands, 43 
combining commands, 44-46 
command history, 48 
command-line editing, 47 
configuring, 58 
defining variables, 39-41 
evaluation prevention, 46-47 
filename completion, 49 
identifying, 37 
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input/output redirection, 
43-44 
invoking directly, 325-327 
job control, 50-53 
killing commands, 57 
pattern matching, 37-39 
printf command, 277 
programming with shell 
scripts, 311 
running as a command, 37 
running multiple shells at 
once, 54-56 
search path, 41-42 
tailoring behavior of, 58 
terminating, 58 
type command, 97, 106 
-bashrc file, 38, 42, 58, 189 
-bash_login file, 58 
-bash_logout file, 56, 58 
-bash_profile file, 58 
bc command, 285-287 
bg command, 51, 52 
bin directory, 32 
binary files, viewing, 77 
block devices, 216 
Booleans in shell scripts, 285, 314 
/boot directory, 34 
braces (see { }) 
broken link, 67 
btop command, 168 
Btrfs, 236 
built-in commands, 43 
bunzip2 command, 130 
burning CDs and DVDs, 246 
bzcat command, 130 
bzip2 command, 130 
software installation and, 209 
tar -j command and, 129 


C 

Cand C++ languages, 327 

AC keystroke (killing programs), 
o/ 


cal command, 23, 289 
calculator programs, 283 
calendar printing, 289 
caret (see 4) 
carriage returns, 112, 117 
case statement, 319 
cat command, x, 72, 118 
gzip command and, 129 
revealing end-of-line charac- 
ters, 112 
split command and, 149 
tee command and, 124 
cd command, 27, 29, 30, 69 
cdparanoia command, 305 
CDs (compact discs) 
burning, 246 
ripping audio, 305 
cgi-bin directory, 32 
chattr command, 96 
checksums 
comparing, 139 
security, 135 
shasum command, 139 
chgrp command, 36, 91, 193 
chmod command, 36, 91-94 
chown command, 36, 90 
chronological order, reversing, 
122 
chsh command, 37, 192 
cksum command, 135 
clear (the screen) command, 24, 
281 
clipboard 
copy and paste actions, 
281-283 
selections, 281 
Wayland window system, 281 
X window system, 281 
clock programs, 289 
cloning hard drives, 245 
cloud storage providers, 243 
cmp command, 135, 138 
Code Industry, 152 
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column command, 115 
columns of text 
appending, 114-115 
creating, 115 
extracting from files, 112-113 
combining commands, 44-46 
comm command, 135, 137-138 
command history, 48 
command mode, vim text editor, 
81 
command prompt, xii 
command substitution, 45, 103, 
170 
command-line arguments, shell 
scripts, 323-324 
command-line editing with bash, 
47 
commands 
built-in, 43 
combined, 44-46 
defined, 19 
examples to try, 22-24 
killing, 57, 170 
previous, 48 
sequences of, 44 
shell scripts, 311-324 
simple, 25 
structure of, 24-26 
compact discs (see CDs) 
comparing files, 135-140 
completing filenames with Tab 
key, 49 
compress command, 129, 131 
compressing/uncompressing files, 
127-133 
compression ratio, ZFS, 239 
conditional execute permission, 
93 
conditionals, shell scripts, 
317-320 
config.fish file, 58 
configuration files, shell, 58 
configure script, 209, 211 


containers, 297-300 
creating, 297 
Docker, 297 
executing, 299 
Flatpak, 206 
images and, 299 
manipulating, 300 
monitoring, 300 
package management, 197 
Snap, 208 
convert command, 302 
copy and paste actions, 281-283 
cp command, 64 
cpio command, 241 
cron process, 178 
crontab command, 178-180 
.cshrc file, 58 
csplit command, 149-151 
CSS selectors, 143 
csvtool command, 147-148 
CUPS printing system, 156 
curl command, 24, 272 
downloading practice files, 
xiv 
piping into bash, 325 
piping into hxselect, 144 
curly braces (see { }) 
current directory, 28, 69 
cut command, 112-113 


D 

AD keystroke (end of file), 58 
logging out, 181 
single-user mode, 183 
superuser shells, 162 
tmux command, 56 

dangling link, 67 

dash (see -) 

dash shell, 58 

dataset, ZFS, 236, 238 

date command, 176, 290-292 

dates and times, displaying and 
setting, 289-292 
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dc command, 287-288 
dd command, 245-246 
deb file, 196 
Debian packages, 196 
apt command, 200 
aptitude command, 201 
dpkg command, 200 
decimal places 
be command, 285 
dc command, 288 
default editor, setting, 79 
defaultdomain file, 269 
/dev directory, 33 
df command, 23, 214-215 
diff command, 45, 135-137 
diff3 command, 135, 137 
dig command, 254 
directories, Linux, 27 
changing, 69 
comparing, 137 
creating, 70 
default mode for creating, 
94-95 
deleting empty, 71 
deleting nonempty, 71 
group ownership, 91 
home directories, 29-30 
kernel-related directories, 
33-35 
ownership, 90, 91 
printing absolute path of, 69 
printing final part of path, 69 
printing path without final 
part, 70 
protection from unauthorized 
users, 91-94 
system directories, 30-33 
dirname command, 70 
disks 
disk usage, 86-87 
logical volume management, 
229-235 
RAID arrays, 224-229 


using, 213-219 
disown command, 53 
disowning jobs, 50 
display command, 301 
DISPLAY environment variable, 
40 
distro, 20 
identifying, 196 
installing software, 196 
dnf command, 196 
DNS (domain name service), 
querying, 253 
doc directory, 32 
Docker, 297-301 
containers 
copying files to/from, 300 
creating, 299 
deleting, 300 
detecting changes, 300 
killing, 299 
listing, 298, 300 
logs, 300 
pausing, 299 
processes, listing, 300 
renaming, 300 
restarting, 299 
running commands in, 
300 
stopping, 299 
Dockerfile, 297 
images, 298-301 
building, 298, 299 
deleting, 300 
downloading, 299 
running, 298 
searching for, 299 
manual, 300 
registry, 298 
docker command, 297-300 
Dockerfile, 297 
dollar sign (see $) 
domain name service (DNS), 
querying, 253 
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dot (see .) 

dot files, 38, 62 

double ampersand (&&), logical 
and, 44 

double dash (--), end of options, 
xii 

double pipe (||), logical or, 44 

downloading files, xiv, 273 

dpkg command, 196, 200, 201 

du command, 86-87 

dump command, 96, 241 

duplicate lines of text, 120-122 


E 
e2label command, 223 
echo command, xiii, 275-276 
built-in versus /bin/echo, 276 
script output, 314 
EDITOR environment variable, 
73,79 
egrep command, 111 
elinks command, 271 
else statement, 318 
emacs text editor, 80 
bash command-line editing, 
47 
creating/editing files, 78 
flatpak command and, 206 
lynx -emacskeys command, 
272 
email, 262-269 
directory, 33, 40 
file format, 266 
log file, 265 
pipelines, 265 
queue, 265 
readers, 262-265 
reading over SSH connection, 
263 
scripting, 264 
servers, 266-269 
emerge command, 202-204 
empty files, creating, 79 


encrypting files, 133-134, 243 
enlarging filesystems, 222 
environment variables, 40 

DISPLAY, 40 

EDITOR, 73, 79 

HOME, 30, 40 

LOGNAME, 40 

MAIL, 40 

OLDPWD, 40 

PATH, 40 

printing, 188 

PWD, 40 

SHELL, 40 

TERM, 40 

USER, 40 

VISUAL, 73, 79 
equals sign (=), variable assign- 

ment, 39 
escape characters 

echo command, 276 

printf command, 278 

tr command, 117 
escaping special characters, 47 
etc directory, 32 
evaluation of special characters, 

preventing, 46-47 
evince command, 152 
examples, running, xiv 
Excel documents, 78 
exclamation point (see !) 
execute permission, 35, 91 
exit codes 

defined, 314 

exiting with, 324 

test command, 315 
exit command, 181, 324 

single-user mode, 183 

terminating shells, 58 
expand command, 118 
export command, 40 
expr command, 284-285 
ext3 filesystems, 96 
ext4 filesystems, 214 
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extended regular expressions, 
108, 111 


F 
fdisk command, 220 
fetchmail command, 266 
ffmpeg command, 309 
fg command, 51, 53 
fgrep command, 111 
file command, 88 
filename completion, 49 
files 
attributes of, 84-97 
case sensitive names, 29 
changing attributes, 96 
comparing, 135-140 
compressing and packaging, 
127-133 
converting, 141-151 
copying, 64, 245 
counting words, 86 
creating, 78-79, 89 
default mode for creating, 
94-95 
deleting, 65 
disk usage, 86-87 
downloading, 273 
encrypting, 133-134, 243 
group ownership, 91 
linking, 66-68 
listing, 61-63 
listing attributes, 84-86, 95-96 
locating, 97-107 
MIME type, 88 
moving, 64 
ownership, 35, 63, 90, 91 
pattern matching, 37 
PDF and PostScript files, 
152-155 
permissions, 35-36, 63, 91, 
104 
protecting in /etc/postfix, 268 


protection from unauthorized 
users, 91-94 
renaming, 64 
reporting type of, 88 
test arguments for, 315 
text editors, 78-81 
text manipulation, 107-124 
timestamps, 89 
transferring between 
machines, 259 
viewing, 71-78 
filesystems, 27-36 
backups and remote storage, 
241-247 
checking integrity, 218 
creating and modifying, 
220-223 
ext4, 214 
file permissions, 35-36 
fixing errors, 218 
home directories, 29-30 
kernel-related directories, 
33-35 
logical volume management, 
229-235 
RAID arrays, 224-229 
system directories, 30-33 
using, 213-219 
ZFS, 235-241 
find command, 97, 98-101, 102 
firewalls 
blocking pings, 255 
blocking traceroute, 256 
fish shell, 58 
FLAC audio file, 307 
flatpak command, 206 
flock command, 174 
folders (see directories) 
fonts directory, 32 
for loops, shell scripts, 322, 324 
foreground jobs, 50, 53 
format specifications 
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format specifications, printf, 
277-278 
formatting disks, 214 
free command, 168-169 
fsck command, 183, 218-219 
fstab file 
lvm2, 232 
mount command, 217 
RAID array, 226 
full name of user 
changing, 190 
setting, 189 


functions, bc command defining, 


286 


G 

g++ command, 327 
gcc command, 327 
GECOS field, 190 

git command, 292-295 


cloning repositories, 210, 293 


diff format, 136 


GNOME graphical environment, 


20, 22 
gnome-terminal command, 22 


GNU emacs (see emacs text edi- 


tor) 
GNU Privacy Guard (GnuPG), 
133: 
gnumeric command, 78 
gparted command, 220, 222 
gpg command, 133-134 


graphical desktop environment, 


20, 21; 56 

graphics (see images) 

grep command, 108-109 
egrep command and, 111 
fgrep command and, 111 
ps command and, 165 
strings command and, 76 

group IDs, 185 

groupadd command, 193 

groupdel command, 194 


groupmod command, 195 
groups, 35, 192-195 


creating, 193 

deleting, 194 

logical volume, 235 

modifying, 195 

ownership of files, 91 

printing group membership, 
193 

updating membership, 191 

using new membership 
immediately, 194 


groups command, 193 
growisofs command, 246 
gs command, 153 

gunzip command, 129 
gv command, 152 

gzip command 


software installation and, 209 
tar -z command and, 129 


gzip compression, 239 


H 


hangup signal, 173 
hard drive cloning, 245 
hard links, 66 
hardware platform, 250 
hash mark 


cron comments, 179 
superuser prompt, 162 


head command, 75 

help and tutorials, 59-60 

--help option, xii, 60, 252 

history command, 48 

home directories, 29-30 

HOME environment variable, 30, 


40 


host command, 253-254 
host information, 249-252 


local, 249 
remote, 253 


host location, 253-256 
hostfile file, 251 
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hostname command, 250-251 

HTML data, extracting strings 
from, 143 

html directory, 32 

hxnormalize command, 144 

hxselect command, 143-144 


| 
ICMP packets, 255 
id command, 185 
ID3 tags 
modifying, 306 
viewing, 306 
id3info command, 306 
id3tag command, 306 
if statement, shell scripts, 317 
ifconfig command, 252 
(see also ip command) 
iftop command, 168 
ImageMagick, 301 
images, 301-303 
combining, 303 
converting, 302 
displaying, 301 
Docker, 299-300 
modifying, 302 
optical character recognition, 
135 
include directory, 32 
index of file locations, creating, 
104 
infinite loops, 322 
info command, 59 
info directory, 32 
init.d directory, 32 
initialization files, shell, 58 
inode, 66, 85 
input redirection (<) operator, 43 
input to a shell script, 314 
input/output redirection, 43-44 
insert mode, vim text editor, 81 
installing software 
from source code, 209-212 


package installation, 196-208 
Internet domains, looking up reg- 
istration of, 254 
ionice command, 172 
iotop command, 168 
ip command, 251-252 
ISO files, 245, 246 
itsfoss.community, 60 


J 

Java language, 327 

javac command, 327 

job control in Linux shells, 50-53 
job scheduling, 175-180 

jobs command, 51 

journalctl command, 182 

jq command, 144-145 

JSON, 116, 144 


K 
k3b command, 247 
Kaffeine media player, 309 
KDE graphical environment, 20, 
22 
kernel, 19 
kernel-related directories, 
33-35 
name and version, 249 
key pair 
GnuPG, 133 
SSH, 258 
kill command, 57, 170-171 
konsole command, 22, 54 
ksh shell, 58 
-kshre file, 58 


L 

labels, filesystem, 223 
lame command, 305 
language codes, 89 
lartc.org website, 252 
last command, 24, 187 
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leading zeros, 280 
less command, x, 72-74 
lib directory, 32 
LibreOffice, 78 
line continuation character, xiii, 
312 
line numbering 
cat -n command, 72 
nl command, 74 
linebreaks 
converting, 112 
grep handling of, 112 
shell scripts, 312 
Windows and MacOS, 112 
links, 66 
creating, 66-68 
hard versus symbolic, 67 
output of ls, 63 
links web browser, 271 
Linux, 19 
(see also commands) 
components of, 19-20 
defined, 19 
help and tutorials, 59-60 
scripting languages, 327 
linuxpocketguide directory, xiv 
linuxquestions.org website, 60 
In command, 66-68 
load average, 166 
locate command, 97, 104-105 
lock directory, 33 
lockfile, 174 
logfiles 
directory, 33 
reversing, 122 
logical volume management 
(LVM), 229-235 
(see also lvm2) 
adding volumes, 232, 233 
creating volumes, 231 
deleting logical volumes, 234 
deleting physical volumes, 
235 


enlarging volumes, 233 
shrinking volumes, 234 
viewing details, 232 
volume groups, 234 

login file, 58 

login shell, 162, 192 

logins/logouts, 181-184 
changing login shell, 192 
listing logged-in users, 186 
logging into remote 

machines, 257 

printing login names, 184 
viewing recent, 187 

logname command, 184, 185 

LOGNAME environment vari- 

able, 40, 184 

logout command, 181 

long command lines, xiii, 312 

look command, 158 

loops, shell scripts, 321-323 

/lost+found directory, 34 

Ipq command, 156, 157 

Ipr command, 24, 156 

Iprm command, 156, 157 

LPRng printing system, 156 

ls command, xi, 23, 61-63 

displaying file attributes, 84 

file permissions, 35 

lsattr command, 95-96 

Isblk command, 216, 225, 233 

lvcreate command, 229, 231, 232 

lvdisplay command, 229, 232 

lvextend command, 233 

LVM (see logical volume manage- 

ment; lvm2) 

lvm2 
logical volumes, 230 
physical volumes, 230 
RAID combined with, 230 
volume group, 230 

Ivremove command, 229, 234 

Ivresize command, 229, 234 
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LVs (see logical volume manage- 
ment; lvm2) 

lvs command, 232 

Lynx web browser, 270-271 


M 
m4 macro processor, 125 
mail (see email) 
mail command, 264-265 
mail directory, 33, 40 
MAIL environment variable, 40 
mail server 
nullmailer, 268 
relay-only, 268 
mailq command, 265 
main.cf file, 267 
make command, 209, 211 
malicious archive file, 210 
man command, 32, 59 
man directory, 32 
masks and file permissions, 95 
master boot record (MBR), 246 
master.cf file, 267 
math commands, 283-288 
md5sum command, 135, 140 
mdadm command 
adding disks, 228 
creating arrays, 224 
destroying arrays, 228 
LVM combined with, 230 
managing arrays, 224 
mdadm.conf file, 226 
removing disks, 227 
replacing disks, 227 
saving configuration, 226 
viewing arrays, 225 
mdstat file, 224 
/media directory, 33 
mediainfo command, 304 
memory usage, displaying, 169 
meta key, 80, 81 
metaflac command, 307 


Microsoft CAB files, extracting, 

132 
Microsoft documents, editing, 78 
MIDI, 304 
MIME 

attachments, 132 

type detection, 88 
mimetype command, 88 
mirroring 

RAID, 224 

rclone, 243 

rsync, 241 

ZFS, 236 
mkdir command, 70 
mke2fs command, 221, 232 
mkfs commands, 221 
mkisofs command, 247 
mkntfs command, 221 
mlocate command, 104 
/mnt directory, 33 
mode 0600, 268 
mogrify command, 302 
mono command, 327 
montage command, 303 
mount command, 217 

lvm2, 232 

RAID, 226 
mounting filesystems, 214 
MOV files, 310 
MP3 files 

creating from WAV files, 305 

extracting audio from MP4 

files into, 310 

ID3 tags, 306 
MP4 files, 310 
mpack command, 132 
mplayer command, 309 
mplayerhgq.hu website, 309 
mtr command, 256 
munpack command, 132 
music (see audio) 
mutt command, 263-264 
mv command, 64, 326 
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N 
nameserver, 253 
nano text editor, 80 
nc command, 261 
-NET, 327 
netcat command, 261-262 
network interface, viewing, 251 
networking, 249-274 
email, 262-265 
email servers, 266-269 
host information, 249-252 
host location, 253-256 
network connections, 257-262 
web browsing, 270-274 
newgrp command, 194 
nice command, 172 
nixcraft.com website, 60 
nl command, 72, 74 
nohup command, 173 
NOT AND binary operation, 95 
nslookup command, 254 
NTES filesystems, 214 
Nullmailer, 268-269 
nullmailer-send command, 269 


0 
OCR (optical character recogni- 
tion), 155 
ocrmypdf command, 155 
octal permissions, 92 
ocular command, 152 
od command, x, 77-78, 118 
Ogg Vorbis audio file, 307 
ogginfo command, 307 
OLDPWD environment variable, 
40 
openSUSE Linux, 205 
OpenZFS Documentation, 236 
operating system 
directories, 33 
printing name of, 250 
optical character recognition 
(OCR), 155 


options for commands, 24 

other (user group), 35 

output from shell scripts, 314 

output redirection (>) operator, 
44 

ownership of files, 35, 63 


P 
package management, 196-208 
apt, 200 
aptitude, 201 
container-based, 197 
dnf, 197 
dpkg, 200 
emerge, 202-204 
flatpak, 206 
pacman, 204 
Portage, 202 
rpm, 199 
snap, 208 
yum, 198 
zypper, 205 
pacman command, 204 
paging keystrokes, 73 
pandoc command, 141-142 
parentheses, 284 
parted command, 220 
partition table, saving, 220 
partitioning a disk, 213, 220 
passwd command, 191 
passwords 
changing, 191 
root, 162 
paste command, 114-115, 326 
patch command, 136 
path 
filesystem, 28 
printing absolute directory 
path, 69 
printing final part, 69 
search, 41-42 
PATH environment variable, 40, 
Al 
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pattern matching (wildcards), 
37-39, 99 
pattern-matching engines, 125 
PDF files, 152-155 
converting from images, 302 
converting from markdown, 
142 
converting to PostScript, 155 
displaying, 152 
extracting attachments into 
files, 132 
manipulating, 154 


optical character recognition, 


155) 
splitting, 153 
text extraction, 152-153 
pdf2ps command, 155 
pdfseparate command, 153 
pdftk command, 154 
pdftotext command, 152 
Perl, 126, 327 
permissions, file, 35-36, 63, 91, 
104 
pgrep command, 165-166, 170 
photos (see images) 
PHP, 126, 327 
physical volumes (PVs), 230 
(see also lvm2) 
PID (see process ID) 
pidof command, 165 
ping command, 255 
pipe (|) operator, 26, 45, 325-327 
pkill command, 170 
plocate command, 104 
pool, ZFS, 236 
Portage package manager, 202 
postfix command, 267 
Postfix mail server, 266-268 
postfix-files file, 267 
postfix.org website, 268 
PostScript files, 152-155 
converting to PDF, 155 
displaying, 152 


text extraction, 153 
Powerpoint documents, 78 
practice files, downloading, xiv 
--prefix option, 211 
printenv command, 178, 188 
printf command, 277-278, 314 
-printf option (find command), 

101 
printing, 156-157 
private key 

GnuPG, 133 

SSH, 258 
/proc directory, 34 
process ID, 163, 165 
process substitution, 45-46 
processes 

controlling, 169-175 

killing, 170-171 

by name, 170 
by PID, 170 

shell jobs versus, 163 

signaling, 170 

viewing, 163-169 
processor type, 250 
procmail command, 266 
-profile file, 58 
prompt, xii 
ps command, 164-165, 170 
ps2ascii command, 153 
ps2pdf command, 155 
public key 

GnuPG, 133 

SSH, 258 
public_html directory, 32 
pvcreate command, 229, 231, 233 
pvdisplay command, 229, 230, 

232 
pvremove command, 229, 235 
PVs (physical volumes), 230 

(see also lvm2) 
pvs command, 229, 231, 232 
pwd command, 27, 69 
PWD environment variable, 40 


Index | 341 


Python, 126, 327 


Q 

question mark (?), filename pat- 
tern, 38 

quoting 
on command line, 47 
in shell scripts, 313 


R 
RAID arrays, 224-229 
(see also mdadm) 
creating, 224-226 
destroying, 228 
drive failure, 227 
mounting, 224, 226 
replacing devices in, 227-228 
RAID level, 224 
rc.d directory, 32 
rclone command, 243-244 
read command, 314 
read permission, 35, 91 
readlink command, 68 
real name of user 
changing, 190 
setting, 189 
reboot command, 183 
rebooting, hostname command 
and, 251 
redirecting input/output, 43-44 
regular expressions 
awk command, 125 
csplit command, 149 
egrep command, 111 
find -regex command, 99 
grep command, 108, 109 
less command, 73 
line numbering with nl, 74 
locate -r command, 105 
pgrep command, 165 
relative path, 28 
relative permissions, 92 


relay-only mail server, 268 
remote machines 
file transfers, 259, 260 
hostname lookup, 253 
logging in with ssh, 257 
sending ICMP packets to, 255 
traceroute command, 255 
remote storage, 241-247 
remotes file, 269 
renice command, 173 
reset command, 57 
reset-lpg script, xiv, 311 
resize2fs command, 222-223, 233 
REST APIs, testing, 272 
restore command, 241 
resuming jobs, 50, 53 
return codes of commands, 314, 
324 
reverse Polish notation (RPN), 
287 
revision IDs, 294, 297 
ripping CD tracks, 305 
rm command, 65, 325 
rmdir command, 71 
root directory (/), 27, 29 
root password, 162 
root user, 26, 161 
(see also superuser) 
rpm command, 196, 199 
rpm file, 196 
RPM packages, 196 
dnf command, 197 
rpm command, 199 
yum command, 198 
zypper command, 205 
RPN (reverse Polish notation), 
287 
rsnapshot command, 242 
rsync command, 174, 241-243 
Ruby, 126, 327 
run directory, 33 
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S 
sasl_passwd file, 267 
sbin directory, 32 
scheduled jobs, running 
at command, 176 
crontab command, 178 
scp command, 259 
screen command, 54, 281 
screen output, 275-281 
sdiff command, 135, 137 
search path, 41-42 
secure shell program, 257 
sed command, 117, 125, 326 
selections (clipboard), 281 
semicolon (;), combined com- 
mands, 44 
seq command, 279-281, 323 
.service suffix, 182 
setgid, 93 
setting dates and times, 289-292 
setuid, 93 
7z (7-Zip) command, 132 
sfdisk command, 220 
sftp command, 260 
sgdisk command, 228 
share directory, 32 
shasum command, 135, 139-140 
shell, xii, 20 
(see also bash shell) 
changing login shell, 192 
history-related commands, 48 
identifying, 37 
job control, 50-53 
launching, 21 
versus program, 43 
prompt, xii 
running multiple shells at 
once, 54-56 
suspending, 52 
tailoring behavior of, 58 
versus terminal, 22 
terminating, 58 
window, 21 


SHELL environment variable, 40 
shell scripts, 311-324 
Booleans, 314 
command-line arguments, 
323-324 
conditionals, 317-320 
creating, 311-312 
exit codes, 314, 324 
input/output, 314-314 
linebreaks, 312 
loops, 321-323 
piping to bash, 325-327 
running, 311-312 
test command, 315-317 
variables, 313 
whitespace, 312 
writing, 311 
shrinking filesystems, 222 
shuf command, 122-124 
shutdown command, 182-183 
signals, 171 
simple command, 25 
single-user mode, 183 
slash (see /) 
sleep command, 171, 175 
slocate command, 104 
snap command, 208 
snapshots, ZFS, 240 
soft links, 67 
software 
installing from source code, 
209-212 
package installation, 196-208 
sort command, 118-120 
sorting key, 119 
sound (see audio) 
source command, 312 
sox command, 308 
spaces, converting to tabs, 118 
SpamAssassin, 266 
special characters, escaping, 47 
spell command, 159 
spelling checkers, 158-159 
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split command, 148-149 
spool directory, 33 
square brackets (see [ ]) 
src directory, 32 
ssh command, 21, 242, 257-258 
ssh-copy-id command, 258 
ssh-keygen command, 258 
Stack Exchange, 60 
stack-based calculator, 287 
staging, 293 
standard error (stderr), 43 
standard input (stdin), xi, 43 
standard output (stdout), xi, 43, 
275. 
startup files, shell, 58 
stat command, 84-86 
storage devices 
erasing, 221 
exporting partition tables of, 
220 
listing all, 216 
listing partitions of, 220 
replacing failed devices, 228 
strings command, 76 
striping disks, 236 
su command, 162, 163 
subdirectories, Linux, 27 
subprocess, 40 
Subversion, 295 
(see also svn command) 
sudo command, 26 
becoming superuser, 161 
versus su command, 163 
whoami command and, 185 
sum command, 135 
superuser, 26 
becoming, 161-163 
useradd command, 189 
suspend command, 52 
suspending jobs, 50 
svn command, 295-297 
symbolic links, 67 
sync command, 96 


/sys directory, 34 
system administration, 161-212 
becoming superuser, 161-163 
controlling processes, 169-175 
group management, 192-195 
installing software from 
source code, 209-212 
installing software packages, 
196-208 
logins/logouts and shut- 
downs, 181-184 
scheduling jobs, 175-180 
user account management, 
188-192 
users and user environment, 
184-188 
viewing processes, 163-169 
system directories, 30-33 
directory path application, 33 
directory path categories, 
31-33 
directory path scope, 31 
system load, 166 
systemctl command, 181-182 
systemd service manager, 181 


T 

Tab key, completing filenames 
with, 49 

table formatting, 115 

tabs, converting to spaces, 118 

tac command, 122 

tail command, 75-76 

tar command, 128-129 
extracting practice files, xiv 
software installation and, 209 

tar files, 209 
bzipped, 130 
compressed, 131 
gzipped, 130 

tarball, 209 

tcsh shell, 58 

.tcshre file, 58 
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tee command, 124 
telnet, 262 
TERM environment variable, 40 
terminal device names, printing, 
187 
terminal program, 21-22 
terminating shells, 58 
tesseract command, 156 
test command, 315-317 
text manipulation commands, 
107-124 
text-based email programs, 262 
tilde (~), home directories, 30 
time, displaying and setting, 
289-292 
timeout command, 171 
timestamps of files 
listing, 85 
modifying, 89 
printing, 63 
tmp directory, 33 
tmux command, 54, 281 
top command, 23, 167-168 
touch command, 79, 89 
tr command, 116-117 
traceroute command, 255-256 
translating characters, 116-117 
tree, 27 
trunk (subversion branch), 296 
tty command, 187 
tutorials, 59-60 
emacs, 80 
Linux help, 59 
vim editor, 81 
type command, 41, 43, 97, 106 
types of files, reporting, 88 


U 

umount command, 218 
unalias command, 62 

uname command, 34, 249-250 
uncompress command, 131 
unexpand command, 118 


uniq command, 120-122 
unix.stackexchange.org website, 
60 
unmask command, 94-95 
until loops, shell scripts, 321-322 
unzip command, 131 
update-initramfs command, 226 
updatedb command, 104 
uptime command, 34, 166 
USER environment variable, 40 
useradd command, 189 
userdel command, 190 
usermod command, 190-191 
users, 26, 184-192 
(see also superuser) 
administrator, 26 
creating accounts, 189 
deleting, 190 
effective username, printing, 
185 
environment variables, 
related, 188 
group membership, printing, 
193 
login names, printing, 184 
login times, listing, 187 
logins, active, 186 
password changes, 191 
protecting files from unau- 
thorized, 91-94 
user IDs, printing, 185 
users command, 186 
/usr/share/doc directory, 59 
uxterm command, 22 


V 
/var directory, 33 
variables 

defining, 39-41 

in shell scripts, 313 
vdev, ZFS, 235 
version control, 292-297 
VG (see volume group) 
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vgcreate command, 229, 230, 231 
vgdisplay command, 229, 232 
vgextend command, 229, 233 
vgreduce command, 229, 234 
vgremove command, 229, 234 
vgs command, 232 
vi (see vim text editor) 
video, 309-310 
viewing 
files, 71-78 
images, 301 
processes, 163 
videos, 309 
vim text editor, 78, 81 
bash command-line editing, 
47 
less command, 73 
lynx -vikeys command, 272 
sed and, 125 
VISUAL environment variable, 
73,79 
VLC media player, 309 
volume group, 230 
(see also lvm2) 


W 
w command, 167 
w3m command, 271 
watch command, 176 
Wayland window system, 20, 281 
wc command, 23, 24, 86 
web browsing, 270 
automation, 273 
retrieving pages via command 
line, 273 
text-based, 270 
wget command, 271, 273-274, 325 
whereis command, 98, 106 
which command, 41, 97, 105 
while loops, shell scripts, 321-322 
whitespace 
arguments separated by, 284 
linebreaks, 112 


programming with shell 
scripts, 312 
quoting on command line, 47 
square brackets (test com- 
mand), 316 
xargs command and, 102 
who command, 105, 186 
whoami command, 162, 185 
whois command, 24, 254 
wildcards (pattern matching), 
37-39, 99 
windows (shell), opening, 21 
wl-copy command, 281 
wl-paste command, 281 
Word documents, 78 
working directory (see current 
directory) 
world (user group), 35 
write permission, 35, 91 
www directory, 32 


X 

X window system, 20 
clipboard commands, 281 
DISPLAY variable, 40 
X11 directory, 32 

xargs command, 101-103 
command substitution, com- 

parison to, 103 

find command and, 102 
null-separated lines, 101 

xclip command, 282 

xine video player, 309 

xmllint command, 145-147 

XPath expression, 146 

xsel command, 283 

xterm command, 22 


yes command, 279 
yum command, 196, 198 
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Z 

AZ keystroke (suspending jobs), 
Sl 

zcat command, 129, 131 

zcmp command, 127 

zdiff command, 127 

zeros, leading, 280 

ZFS (Zettabyte File System), 
235-241 
creating datasets, 238 
creating encrypted datasets, 

239 

creating pools, 237-238 
destroying datasets, 240 
destroying pools, 241 
destroying snapshots, 240 
enabling compression, 239 


setting size limits on datasets, 
239 
snapshotting datasets, 240 
zfs command, 235 
zfs destroy command, 240 
zgrep command, 127 
zip command, 131, 209 
zless command, 127 
.Zlogin file, 58 
.Zlogout file, 58 
zpool command, 235, 237 
.zprofile file, 58 
.zshenv file, 58 
.zshrc file, 58 
zstd command, 204 
zypper command, 196, 205 
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